[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:25:53 dist-test-slave-bt27 krb5kdc[5834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:25:53 dist-test-slave-bt27 krb5kdc[5834](info): set up 2 sockets
Jun 19 22:25:53 dist-test-slave-bt27 krb5kdc[5834](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 22:25:54.722347 5827 mini_kdc.cc:121] Time spent starting KDC: real 3.161s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 22:25:54.975256 5827 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.252s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:25:55.106714 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34157
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:34157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:55.232318 5847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:55.232729 5847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:55.232776 5847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:55.236927 5847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:25:55.237003 5847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:55.237025 5847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:55.237043 5847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:25:55.237059 5847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:25:55.242704 5847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:34157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:55.243906 5847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:55.245000 5847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:55.254901 5855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:55.254948 5847 server_base.cc:1034] running on GCE node
W20260619 22:25:55.254935 5853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:55.255117 5852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:55.255812 5847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:55.257056 5847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:55.258252 5847 hybrid_clock.cc:648] HybridClock initialized: now 1781907955258171 us; error 97 us; skew 500 ppm
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:55.261286 5847 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:25:55.262313 5847 webserver.cc:460] Webserver started at http://127.5.176.254:35333/ using document root <none> and password file <none>
I20260619 22:25:55.262935 5847 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:55.263031 5847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:55.263273 5847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:55.265460 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "5213d64c5fd04af3a81b36a930d14fa1"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "d7d949211aa33c55028d21c6e3c39055"
server_key_iv: "562118ea98bb3eb50c21e9e91d08afa8"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.266083 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "5213d64c5fd04af3a81b36a930d14fa1"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "d7d949211aa33c55028d21c6e3c39055"
server_key_iv: "562118ea98bb3eb50c21e9e91d08afa8"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.270427 5847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:25:55.273211 5862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.274802 5847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:25:55.275004 5847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "5213d64c5fd04af3a81b36a930d14fa1"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "d7d949211aa33c55028d21c6e3c39055"
server_key_iv: "562118ea98bb3eb50c21e9e91d08afa8"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.275143 5847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:55.295964 5847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:55.299607 5847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:55.299860 5847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:55.307291 5847 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:34157
I20260619 22:25:55.307348 5914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:34157 every 8 connection(s)
I20260619 22:25:55.308372 5847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:25:55.311640 5915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:25:55.314940 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 5847
I20260619 22:25:55.315129 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:25:55.316877 5827 external_mini_cluster.cc:1428] Setting key fdf3630b3089167f28a70becc9e9ba7f
I20260619 22:25:55.319023 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1: Bootstrap starting.
I20260619 22:25:55.321998 5915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1: Neither blocks nor log segments found. Creating new log.
I20260619 22:25:55.322990 5915 log.cc:824] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1: Log is configured to *not* fsync() on all Append() calls
I20260619 22:25:55.325373 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1: No bootstrap required, opened a new log
I20260619 22:25:55.330559 5915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213d64c5fd04af3a81b36a930d14fa1" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34157 } }
I20260619 22:25:55.330865 5915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:25:55.330951 5915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5213d64c5fd04af3a81b36a930d14fa1, State: Initialized, Role: FOLLOWER
I20260619 22:25:55.331466 5915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213d64c5fd04af3a81b36a930d14fa1" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34157 } }
I20260619 22:25:55.331593 5915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:25:55.331682 5915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:25:55.331809 5915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:25:55.333076 5915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213d64c5fd04af3a81b36a930d14fa1" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34157 } }
I20260619 22:25:55.333479 5915 leader_election.cc:302] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5213d64c5fd04af3a81b36a930d14fa1; no voters:
I20260619 22:25:55.333889 5915 leader_election.cc:288] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:25:55.333964 5920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:25:55.334251 5920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [term 1 LEADER]: Becoming Leader. State: Replica: 5213d64c5fd04af3a81b36a930d14fa1, State: Running, Role: LEADER
I20260619 22:25:55.334599 5920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213d64c5fd04af3a81b36a930d14fa1" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34157 } }
I20260619 22:25:55.335429 5915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:25:55.336345 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5213d64c5fd04af3a81b36a930d14fa1. Latest consensus state: current_term: 1 leader_uuid: "5213d64c5fd04af3a81b36a930d14fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213d64c5fd04af3a81b36a930d14fa1" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34157 } } }
I20260619 22:25:55.336489 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [sys.catalog]: This master's current role is: LEADER
I20260619 22:25:55.336787 5921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5213d64c5fd04af3a81b36a930d14fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213d64c5fd04af3a81b36a930d14fa1" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34157 } } }
I20260619 22:25:55.336877 5921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1 [sys.catalog]: This master's current role is: LEADER
I20260619 22:25:55.337144 5929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:25:55.340531 5929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:25:55.347309 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:55.321082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.5.176.254:34157)
0619 22:25:55.321611 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:55.321631 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:55.322766 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 22:25:55.326865 (+ 4099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:55.326882 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:55.327227 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:55.327502 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:55.330722 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.332078 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.333833 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.334591 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.334763 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:55.340208 (+ 5445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:55.340214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:55.340236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:55.340272 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:55.342774 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:55.343502 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:55.343509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:55.343514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:55.343654 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:55.344079 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:55.344083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:55.344084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:55.344613 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:55.344813 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:55.345570 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 22:25:55.345881 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":149,"threads_started":1}
I20260619 22:25:55.350253 5929 catalog_manager.cc:1347] Generated new cluster ID: 9d8b53cf24d84e4da0633a317a122459
I20260619 22:25:55.350324 5929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:25:55.361604 5929 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:25:55.362919 5929 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:25:55.375216 5929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5213d64c5fd04af3a81b36a930d14fa1: Generated new TSK 0
I20260619 22:25:55.376098 5929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:25:55.410425 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34157
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:55.523276 5941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:55.523638 5941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:55.523723 5941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:55.527932 5941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:55.528033 5941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:55.528123 5941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:25:55.533597 5941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:55.534822 5941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:55.535949 5941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:55.546619 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:55.546643 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:55.546633 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:55.546936 5941 server_base.cc:1034] running on GCE node
I20260619 22:25:55.547365 5941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:55.548049 5941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:55.549213 5941 hybrid_clock.cc:648] HybridClock initialized: now 1781907955549193 us; error 31 us; skew 500 ppm
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:55.552245 5941 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:25:55.553165 5941 webserver.cc:460] Webserver started at http://127.5.176.193:44241/ using document root <none> and password file <none>
I20260619 22:25:55.553776 5941 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:55.553869 5941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:55.554109 5941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:55.556198 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "caf94b87a9684b51a33fe4e88d51b785"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "4c5d540e2e4e943d67650c3807e819a9"
server_key_iv: "792dcaa28923b81b3997f28eae340e1a"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.556782 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "caf94b87a9684b51a33fe4e88d51b785"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "4c5d540e2e4e943d67650c3807e819a9"
server_key_iv: "792dcaa28923b81b3997f28eae340e1a"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.560842 5941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 22:25:55.563309 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.564543 5941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:25:55.564685 5941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "caf94b87a9684b51a33fe4e88d51b785"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "4c5d540e2e4e943d67650c3807e819a9"
server_key_iv: "792dcaa28923b81b3997f28eae340e1a"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.564800 5941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:55.584481 5941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:55.587939 5941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:55.588194 5941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:55.588899 5941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:25:55.589838 5941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:25:55.589917 5941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.589989 5941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:25:55.590046 5941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.599980 5941 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:45315
I20260619 22:25:55.600005 6069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:45315 every 8 connection(s)
I20260619 22:25:55.601003 5941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:25:55.607525 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 5941
I20260619 22:25:55.607659 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:25:55.607910 5827 external_mini_cluster.cc:1428] Setting key 66777e240464be174d4f26122dc23383
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:25:55.619135 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:55.603370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:44999 (local address 127.5.176.254:34157)
0619 22:25:55.603564 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:55.603572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:55.604574 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 22:25:55.606807 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:55.606811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:55.606863 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:55.606943 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:55.608811 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.609421 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.610805 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.611076 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.611199 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:55.614331 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:55.614339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:55.614346 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:55.614384 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:55.616378 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:55.617008 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:55.617015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:55.617018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:55.617130 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:55.617576 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:55.617580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:55.617582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:55.617890 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:55.618015 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:55.618698 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 22:25:55.618893 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:25:55.619964 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:55.603777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34157 (local address 127.5.176.193:44999)
0619 22:25:55.604353 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:55.604426 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:55.606490 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:55.607181 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:55.607190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:55.607799 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:25:55.608625 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:55.608650 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.609566 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.609570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:55.610669 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:55.610679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.611255 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.611259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:55.611459 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:55.612143 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:25:55.612171 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:25:55.614079 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:25:55.616533 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:55.616539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:55.616554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:55.616862 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:55.617263 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:55.617266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:55.617267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:55.617436 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:55.618026 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:25:55.618032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:25:55.618405 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 22:25:55.618584 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 22:25:55.618951 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":162,"threads_started":1}
I20260619 22:25:55.621495 6070 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:25:55.621852 6070 heartbeater.cc:464] Registering TS with master...
I20260619 22:25:55.622454 6070 heartbeater.cc:511] Master 127.5.176.254:34157 requested a full tablet report, sending...
I20260619 22:25:55.624245 5879 ts_manager.cc:194] Registered new tserver with Master: caf94b87a9684b51a33fe4e88d51b785 (127.5.176.193:45315)
I20260619 22:25:55.625452 5879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:44999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:25:55.642016 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34157
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:55.756911 6075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:55.757237 6075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:55.757284 6075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:55.761732 6075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:55.761806 6075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:55.761876 6075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:25:55.767419 6075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:55.768623 6075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:55.769790 6075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:55.779974 6083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:55.779969 6081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:55.780072 6080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:55.780311 6075 server_base.cc:1034] running on GCE node
I20260619 22:25:55.780761 6075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:55.781445 6075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:55.782636 6075 hybrid_clock.cc:648] HybridClock initialized: now 1781907955782615 us; error 35 us; skew 500 ppm
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:55.789577 6075 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:25:55.790747 6075 webserver.cc:460] Webserver started at http://127.5.176.194:44045/ using document root <none> and password file <none>
I20260619 22:25:55.791357 6075 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:55.791452 6075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:55.791692 6075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:55.793851 6075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "6ef7bebc000047f59460d18cefcaf080"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "884fd10f83db2b41fa05acd75597c106"
server_key_iv: "c6c2ae07ef8c63668214c4fa5646a966"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.794486 6075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "6ef7bebc000047f59460d18cefcaf080"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "884fd10f83db2b41fa05acd75597c106"
server_key_iv: "c6c2ae07ef8c63668214c4fa5646a966"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.798473 6075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:25:55.800985 6090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.802601 6075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:25:55.802791 6075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "6ef7bebc000047f59460d18cefcaf080"
format_stamp: "Formatted at 2026-06-19 22:25:55 on dist-test-slave-bt27"
server_key: "884fd10f83db2b41fa05acd75597c106"
server_key_iv: "c6c2ae07ef8c63668214c4fa5646a966"
server_key_version: "encryptionkey@0"
I20260619 22:25:55.802933 6075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:55.826612 6075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:55.829610 6075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:55.829823 6075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:55.830497 6075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:25:55.831328 6075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:25:55.831406 6075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.831471 6075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:25:55.831504 6075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:55.840590 6075 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:46479
I20260619 22:25:55.840667 6203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:46479 every 8 connection(s)
I20260619 22:25:55.841567 6075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:25:55.850200 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6075
I20260619 22:25:55.850358 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:25:55.850652 5827 external_mini_cluster.cc:1428] Setting key a265fb25a9f1016bd02f86fd7fbdeb2c
Jun 19 22:25:55 dist-test-slave-bt27 krb5kdc[5834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:25:55.857218 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:55.843510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:47937 (local address 127.5.176.254:34157)
0619 22:25:55.843665 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:55.843671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:55.844738 (+ 1067us) server_negotiation.cc:407] Connection header received
0619 22:25:55.846369 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:55.846372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:55.846433 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:55.846502 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:55.848060 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.848604 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.849875 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.850094 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.850270 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:55.853178 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:55.853182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:55.853185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:55.853210 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:55.855085 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:55.855603 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:55.855609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:55.855611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:55.855696 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:55.856056 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:55.856059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:55.856061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:55.856281 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:55.856426 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:55.856934 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 22:25:55.857057 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:25:55.858681 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:55.844022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34157 (local address 127.5.176.194:47937)
0619 22:25:55.844546 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:55.844602 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:55.846031 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:55.846758 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:55.846780 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:55.847349 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:25:55.847905 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:55.847920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.848729 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.848732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:55.849741 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:55.849751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:55.850352 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:55.850358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:55.850599 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:55.851396 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:25:55.851414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:25:55.852998 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:25:55.855202 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:55.855207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:55.855224 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:55.855487 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:55.855789 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:55.855794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:55.855796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:55.855947 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:55.856422 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:25:55.856428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:25:55.856693 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 22:25:55.856924 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 22:25:55.857278 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260619 22:25:55.861145 6204 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34157
I20260619 22:25:55.861541 6204 heartbeater.cc:464] Registering TS with master...
I20260619 22:25:55.862301 6204 heartbeater.cc:511] Master 127.5.176.254:34157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:25:55.864012 5879 ts_manager.cc:194] Registered new tserver with Master: 6ef7bebc000047f59460d18cefcaf080 (127.5.176.194:46479)
I20260619 22:25:55.865032 5879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:47937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:25:55.886087 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34157
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:56.004912 6209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:56.005199 6209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:56.005244 6209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:56.009600 6209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:56.009677 6209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:56.009783 6209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:25:56.015529 6209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:56.016695 6209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:56.017771 6209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:56.027458 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:56.027447 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:56.027614 6209 server_base.cc:1034] running on GCE node
W20260619 22:25:56.027448 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:56.028149 6209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:56.028817 6209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:56.029996 6209 hybrid_clock.cc:648] HybridClock initialized: now 1781907956029961 us; error 46 us; skew 500 ppm
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:56.032860 6209 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:25:56.033783 6209 webserver.cc:460] Webserver started at http://127.5.176.195:41229/ using document root <none> and password file <none>
I20260619 22:25:56.034375 6209 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:56.034446 6209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:56.034614 6209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:56.036636 6209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "ebe348c48fe3469b977183db6c419acb"
format_stamp: "Formatted at 2026-06-19 22:25:56 on dist-test-slave-bt27"
server_key: "42b96d3ae6d444206a5bbb0a35a114cf"
server_key_iv: "2423fbe80ba84720fb916ab266563d9c"
server_key_version: "encryptionkey@0"
I20260619 22:25:56.037281 6209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "ebe348c48fe3469b977183db6c419acb"
format_stamp: "Formatted at 2026-06-19 22:25:56 on dist-test-slave-bt27"
server_key: "42b96d3ae6d444206a5bbb0a35a114cf"
server_key_iv: "2423fbe80ba84720fb916ab266563d9c"
server_key_version: "encryptionkey@0"
I20260619 22:25:56.040895 6209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:25:56.043315 6224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:56.044539 6209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:25:56.044660 6209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "ebe348c48fe3469b977183db6c419acb"
format_stamp: "Formatted at 2026-06-19 22:25:56 on dist-test-slave-bt27"
server_key: "42b96d3ae6d444206a5bbb0a35a114cf"
server_key_iv: "2423fbe80ba84720fb916ab266563d9c"
server_key_version: "encryptionkey@0"
I20260619 22:25:56.044765 6209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:56.055065 6209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:56.058094 6209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:56.058375 6209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:56.058961 6209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:25:56.059759 6209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:25:56.059808 6209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:56.059846 6209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:25:56.059906 6209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:56.070559 6209 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:37613
I20260619 22:25:56.070598 6337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:37613 every 8 connection(s)
I20260619 22:25:56.071463 6209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:25:56.073953 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6209
I20260619 22:25:56.074075 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:25:56.074401 5827 external_mini_cluster.cc:1428] Setting key 68934710ccfe6e0a407191201f8b3ee5
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[5834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:25:56.086074 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:56.073320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:51661 (local address 127.5.176.254:34157)
0619 22:25:56.073459 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:56.073463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:56.074348 (+ 885us) server_negotiation.cc:407] Connection header received
0619 22:25:56.076003 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:56.076006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:56.076068 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:56.076157 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:56.077477 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.077991 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.079425 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.079716 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.079801 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:56.082507 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:56.082510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:56.082512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:56.082538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:56.084052 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:56.084573 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:56.084576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:56.084577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:56.084654 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:56.084993 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:56.084994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:56.084996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:56.085163 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:56.085237 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:56.085855 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 22:25:56.085955 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:25:56.087064 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:56.073616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34157 (local address 127.5.176.195:51661)
0619 22:25:56.074156 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:56.074225 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:56.075750 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:56.076343 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:56.076350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:56.076725 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:25:56.077340 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:56.077356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.078114 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.078117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:56.079307 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:56.079317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.079891 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.079894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:56.080096 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:56.080692 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:25:56.080708 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:25:56.082335 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:25:56.084158 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:56.084163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:56.084176 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:56.084424 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:56.084746 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:56.084748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:56.084750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:56.084903 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:56.085293 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:25:56.085299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:25:56.085569 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 22:25:56.085809 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 22:25:56.086100 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260619 22:25:56.088287 6338 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34157
I20260619 22:25:56.088548 6338 heartbeater.cc:464] Registering TS with master...
I20260619 22:25:56.089035 6338 heartbeater.cc:511] Master 127.5.176.254:34157 requested a full tablet report, sending...
I20260619 22:25:56.090029 5879 ts_manager.cc:194] Registered new tserver with Master: ebe348c48fe3469b977183db6c419acb (127.5.176.195:37613)
I20260619 22:25:56.090754 5879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:51661
I20260619 22:25:56.101294 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[5834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:25:56.129199 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:56.119404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.5.176.193:45315)
0619 22:25:56.119708 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:56.119726 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:56.119756 (+ 30us) server_negotiation.cc:407] Connection header received
0619 22:25:56.119938 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:56.119950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:56.120147 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:56.120328 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:56.121046 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.121817 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.122727 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.123106 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.123230 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:56.125763 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:56.125772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:56.125786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:56.125809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:56.127371 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:56.127830 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:56.127841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:56.127843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:56.127920 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:56.128205 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:56.128210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:56.128212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:56.128470 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:56.128711 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:56.128870 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 22:25:56.129026 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
W20260619 22:25:56.130292 5984 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54120
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[5834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[5834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:25:56.159317 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:56.148766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.5.176.193:45315)
0619 22:25:56.148948 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:56.148953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:56.149122 (+ 169us) server_negotiation.cc:407] Connection header received
0619 22:25:56.149288 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:56.149296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:56.149340 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:56.149434 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:56.150246 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.150763 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.151608 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:56.151847 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:56.151957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:56.156083 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:56.156087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:56.156089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:56.156121 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:56.157651 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:56.158206 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:56.158213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:56.158218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:56.158320 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:56.158565 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:56.158568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:56.158569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:56.158781 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:56.158891 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:56.159083 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 22:25:56.159188 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 22:25:56.160490 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 5941
I20260619 22:25:56.166193 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6075
I20260619 22:25:56.173332 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6209
I20260619 22:25:56.178922 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 5847
2026-06-19T22:25:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4650 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[6352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[6352](info): set up 2 sockets
Jun 19 22:25:56 dist-test-slave-bt27 krb5kdc[6352](info): commencing operation
krb5kdc: starting...
W20260619 22:25:57.229900 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:25:57 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:25:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:25:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:25:57.353632 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39797
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:57.465178 6365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:57.465515 6365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:57.465598 6365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:57.469795 6365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:25:57.469899 6365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:57.469939 6365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:57.469973 6365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:25:57.470005 6365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:25:57.475474 6365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:39797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:57.476593 6365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:57.477653 6365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:57.486002 6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:57.486037 6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:57.486011 6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:57.486366 6365 server_base.cc:1034] running on GCE node
I20260619 22:25:57.487025 6365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:57.488067 6365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:57.489267 6365 hybrid_clock.cc:648] HybridClock initialized: now 1781907957489250 us; error 36 us; skew 500 ppm
Jun 19 22:25:57 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907957, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:57.497136 6365 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:25:57.498100 6365 webserver.cc:460] Webserver started at http://127.5.176.254:32867/ using document root <none> and password file <none>
I20260619 22:25:57.498711 6365 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:57.498811 6365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:57.499058 6365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:57.501139 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "df0d589d30cc4765af64ec42d05a5996"
format_stamp: "Formatted at 2026-06-19 22:25:57 on dist-test-slave-bt27"
server_key: "913284608451e0380501fd63c1781216"
server_key_iv: "c9a4ddf9eb3f7ace1859e27cd13302ad"
server_key_version: "encryptionkey@0"
I20260619 22:25:57.501706 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "df0d589d30cc4765af64ec42d05a5996"
format_stamp: "Formatted at 2026-06-19 22:25:57 on dist-test-slave-bt27"
server_key: "913284608451e0380501fd63c1781216"
server_key_iv: "c9a4ddf9eb3f7ace1859e27cd13302ad"
server_key_version: "encryptionkey@0"
I20260619 22:25:57.505481 6365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 22:25:57.507994 6380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:57.509305 6365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 22:25:57.509462 6365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "df0d589d30cc4765af64ec42d05a5996"
format_stamp: "Formatted at 2026-06-19 22:25:57 on dist-test-slave-bt27"
server_key: "913284608451e0380501fd63c1781216"
server_key_iv: "c9a4ddf9eb3f7ace1859e27cd13302ad"
server_key_version: "encryptionkey@0"
I20260619 22:25:57.509582 6365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:57.540357 6365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:57.543375 6365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:57.543588 6365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:57.550895 6365 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:39797
I20260619 22:25:57.550915 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:39797 every 8 connection(s)
I20260619 22:25:57.551868 6365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:25:57.554651 6433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:25:57.560253 6433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996: Bootstrap starting.
I20260619 22:25:57.560781 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6365
I20260619 22:25:57.560894 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:25:57.561156 5827 external_mini_cluster.cc:1428] Setting key bb18ae4aae7bca122f2bd749eb52383c
I20260619 22:25:57.563122 6433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996: Neither blocks nor log segments found. Creating new log.
I20260619 22:25:57.563903 6433 log.cc:824] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996: Log is configured to *not* fsync() on all Append() calls
I20260619 22:25:57.565837 6433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996: No bootstrap required, opened a new log
I20260619 22:25:57.569604 6433 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d589d30cc4765af64ec42d05a5996" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39797 } }
I20260619 22:25:57.569900 6433 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:25:57.569952 6433 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d589d30cc4765af64ec42d05a5996, State: Initialized, Role: FOLLOWER
Jun 19 22:25:57 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:25:57.570495 6433 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d589d30cc4765af64ec42d05a5996" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39797 } }
I20260619 22:25:57.570667 6433 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:25:57.570745 6433 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:25:57.570848 6433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:25:57.571964 6433 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d589d30cc4765af64ec42d05a5996" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39797 } }
I20260619 22:25:57.572324 6433 leader_election.cc:302] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0d589d30cc4765af64ec42d05a5996; no voters:
I20260619 22:25:57.572660 6433 leader_election.cc:288] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:25:57.572814 6438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:25:57.573040 6438 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [term 1 LEADER]: Becoming Leader. State: Replica: df0d589d30cc4765af64ec42d05a5996, State: Running, Role: LEADER
I20260619 22:25:57.573374 6438 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d589d30cc4765af64ec42d05a5996" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39797 } }
I20260619 22:25:57.573966 6433 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:25:57.575345 6440 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0d589d30cc4765af64ec42d05a5996. Latest consensus state: current_term: 1 leader_uuid: "df0d589d30cc4765af64ec42d05a5996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d589d30cc4765af64ec42d05a5996" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39797 } } }
I20260619 22:25:57.575462 6440 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [sys.catalog]: This master's current role is: LEADER
I20260619 22:25:57.575399 6439 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0d589d30cc4765af64ec42d05a5996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d589d30cc4765af64ec42d05a5996" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39797 } } }
I20260619 22:25:57.575523 6439 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996 [sys.catalog]: This master's current role is: LEADER
I20260619 22:25:57.575877 6447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:25:57.577562 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:57.562608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.5.176.254:39797)
0619 22:25:57.563061 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:57.563077 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:57.563112 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:25:57.563880 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:57.563900 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:57.564299 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:57.564713 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:57.565503 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:57.566676 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:57.567542 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:57.568053 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:57.568219 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:57.571067 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:57.571072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:57.571085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:57.571112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:57.573264 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:57.573865 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:57.573871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:57.573877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:57.574025 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:57.574432 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:57.574438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:57.574441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:57.574971 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:57.575943 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:57.576101 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 22:25:57.576565 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":134,"threads_started":1}
I20260619 22:25:57.579337 6447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:25:57.585317 6447 catalog_manager.cc:1347] Generated new cluster ID: c546ec6c7b784ed59bc7c6a66cc50db1
I20260619 22:25:57.585510 6447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:25:57.600986 6447 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:25:57.602959 6447 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:25:57.612692 6447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0d589d30cc4765af64ec42d05a5996: Generated new TSK 0
I20260619 22:25:57.613586 6447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:25:57.653715 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:39797
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:57.785609 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:57.785930 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:57.785976 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:57.790228 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:57.790303 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:57.790422 6459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:25:57.796057 6459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:39797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:57.797425 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:57.798609 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:57.808827 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:57.808861 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:57.808861 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:57.809088 6459 server_base.cc:1034] running on GCE node
I20260619 22:25:57.809652 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:57.810425 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:57.811609 6459 hybrid_clock.cc:648] HybridClock initialized: now 1781907957811586 us; error 33 us; skew 500 ppm
Jun 19 22:25:57 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907957, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:57.814741 6459 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:25:57.815774 6459 webserver.cc:460] Webserver started at http://127.5.176.193:36989/ using document root <none> and password file <none>
I20260619 22:25:57.816385 6459 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:57.816480 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:57.816705 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:57.819024 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "afcf95d89b984c47880d89fa3dda844f"
format_stamp: "Formatted at 2026-06-19 22:25:57 on dist-test-slave-bt27"
server_key: "5d22916158e41dde3a818ef77862ee96"
server_key_iv: "b2933bbf244f95220554a4a665f44f23"
server_key_version: "encryptionkey@0"
I20260619 22:25:57.819633 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "afcf95d89b984c47880d89fa3dda844f"
format_stamp: "Formatted at 2026-06-19 22:25:57 on dist-test-slave-bt27"
server_key: "5d22916158e41dde3a818ef77862ee96"
server_key_iv: "b2933bbf244f95220554a4a665f44f23"
server_key_version: "encryptionkey@0"
I20260619 22:25:57.823616 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:25:57.826359 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:57.827732 6459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:25:57.827894 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "afcf95d89b984c47880d89fa3dda844f"
format_stamp: "Formatted at 2026-06-19 22:25:57 on dist-test-slave-bt27"
server_key: "5d22916158e41dde3a818ef77862ee96"
server_key_iv: "b2933bbf244f95220554a4a665f44f23"
server_key_version: "encryptionkey@0"
I20260619 22:25:57.828032 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:57.841661 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:57.845444 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:57.845665 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:57.846367 6459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:25:57.847244 6459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:25:57.847325 6459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:57.847395 6459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:25:57.847427 6459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:57.856590 6459 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:39509
I20260619 22:25:57.856609 6587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:39509 every 8 connection(s)
I20260619 22:25:57.857569 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:25:57.861299 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6459
I20260619 22:25:57.861435 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:25:57.861713 5827 external_mini_cluster.cc:1428] Setting key 7708bb4b72ce37f410aba4dd5248c4bc
Jun 19 22:25:57 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907957, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:25:57.873309 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:57.859523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47659 (local address 127.5.176.254:39797)
0619 22:25:57.859675 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:57.859679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:57.860625 (+ 946us) server_negotiation.cc:407] Connection header received
0619 22:25:57.862447 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:57.862451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:57.862511 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:57.862596 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:57.864122 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:57.864689 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:57.866006 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:57.866488 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:57.866613 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:57.869421 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:57.869425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:57.869431 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:57.869465 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:57.871247 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:57.871760 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:57.871763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:57.871765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:57.871872 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:57.872240 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:57.872243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:57.872244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:57.872420 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:57.872499 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:57.873067 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 22:25:57.873179 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:25:57.874307 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:57.859939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:39797 (local address 127.5.176.193:47659)
0619 22:25:57.860442 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:57.860489 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:57.862197 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:57.862812 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:57.862820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:57.863204 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:25:57.863948 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:57.863973 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:57.864843 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:57.864849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:57.865875 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:57.865885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:57.866632 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:57.866635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:57.866780 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:57.867482 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:25:57.867526 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:25:57.869236 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:25:57.871376 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:57.871381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:57.871395 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:57.871654 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:57.871981 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:57.871984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:57.871986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:57.872135 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:57.872519 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:25:57.872524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:25:57.872777 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 22:25:57.872962 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 22:25:57.873232 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:25:57.875644 6588 heartbeater.cc:347] Connected to a master server at 127.5.176.254:39797
I20260619 22:25:57.875907 6588 heartbeater.cc:464] Registering TS with master...
I20260619 22:25:57.876458 6588 heartbeater.cc:511] Master 127.5.176.254:39797 requested a full tablet report, sending...
I20260619 22:25:57.877990 6397 ts_manager.cc:194] Registered new tserver with Master: afcf95d89b984c47880d89fa3dda844f (127.5.176.193:39509)
I20260619 22:25:57.879153 6397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:47659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:25:57.890008 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:39797
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:58.003855 6593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:58.004170 6593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:58.004216 6593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:58.008652 6593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:58.008729 6593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:58.008793 6593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:25:58.014568 6593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:39797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:58.015806 6593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:58.017024 6593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:58.027837 6598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:58.027881 6601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:58.027956 6593 server_base.cc:1034] running on GCE node
W20260619 22:25:58.027894 6599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:58.028512 6593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:58.029284 6593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:58.030469 6593 hybrid_clock.cc:648] HybridClock initialized: now 1781907958030442 us; error 37 us; skew 500 ppm
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:58.033619 6593 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:25:58.034687 6593 webserver.cc:460] Webserver started at http://127.5.176.194:44883/ using document root <none> and password file <none>
I20260619 22:25:58.035287 6593 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:58.035351 6593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:58.035578 6593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:58.037964 6593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "e1f6ba2cf9bb4622b386341c67982d54"
format_stamp: "Formatted at 2026-06-19 22:25:58 on dist-test-slave-bt27"
server_key: "b40eb26de9b5632c1fd1a2028ffd53bc"
server_key_iv: "04654fcb1f6af9bf5a32ef0b32a1e6b3"
server_key_version: "encryptionkey@0"
I20260619 22:25:58.038669 6593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "e1f6ba2cf9bb4622b386341c67982d54"
format_stamp: "Formatted at 2026-06-19 22:25:58 on dist-test-slave-bt27"
server_key: "b40eb26de9b5632c1fd1a2028ffd53bc"
server_key_iv: "04654fcb1f6af9bf5a32ef0b32a1e6b3"
server_key_version: "encryptionkey@0"
I20260619 22:25:58.042598 6593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:25:58.045336 6608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:58.046952 6593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:25:58.047060 6593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "e1f6ba2cf9bb4622b386341c67982d54"
format_stamp: "Formatted at 2026-06-19 22:25:58 on dist-test-slave-bt27"
server_key: "b40eb26de9b5632c1fd1a2028ffd53bc"
server_key_iv: "04654fcb1f6af9bf5a32ef0b32a1e6b3"
server_key_version: "encryptionkey@0"
I20260619 22:25:58.047222 6593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:58.064123 6593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:58.067930 6593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:58.068147 6593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:58.068794 6593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:25:58.069674 6593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:25:58.069722 6593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:58.069764 6593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:25:58.069780 6593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:58.079953 6593 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:45709
I20260619 22:25:58.079972 6721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:45709 every 8 connection(s)
I20260619 22:25:58.080987 6593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:25:58.086728 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6593
I20260619 22:25:58.086835 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:25:58.087102 5827 external_mini_cluster.cc:1428] Setting key 9e249847c39f490635fb8828a5d77996
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:25:58.097937 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.083081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:46869 (local address 127.5.176.254:39797)
0619 22:25:58.083225 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.083228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.084220 (+ 992us) server_negotiation.cc:407] Connection header received
0619 22:25:58.085727 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.085731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.085796 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.085918 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:58.087418 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.088128 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.090007 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.090321 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.090499 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.093594 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:58.093598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:58.093601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:58.093635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:58.095221 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.095930 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.095933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.095935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.096019 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.096508 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.096511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.096512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.096707 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:58.096788 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.097573 (+ 785us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.097720 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:25:58.098934 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.083512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:39797 (local address 127.5.176.194:46869)
0619 22:25:58.084029 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.084081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.085449 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.086104 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.086112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.086600 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:25:58.087254 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.087272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.088289 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.088292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.089853 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.089869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.090541 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.090548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.090746 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.091437 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:25:58.091454 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:25:58.093385 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:25:58.095368 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:58.095380 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:58.095401 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:58.095800 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:58.096128 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:58.096135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:58.096138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:58.096397 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:58.096830 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:25:58.096839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:25:58.097230 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.097511 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.097926 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260619 22:25:58.100320 6722 heartbeater.cc:347] Connected to a master server at 127.5.176.254:39797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:25:58.100610 6722 heartbeater.cc:464] Registering TS with master...
I20260619 22:25:58.101193 6722 heartbeater.cc:511] Master 127.5.176.254:39797 requested a full tablet report, sending...
I20260619 22:25:58.102451 6397 ts_manager.cc:194] Registered new tserver with Master: e1f6ba2cf9bb4622b386341c67982d54 (127.5.176.194:45709)
I20260619 22:25:58.103078 6397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:46869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:25:58.118196 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:39797
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:58.233026 6727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:58.233314 6727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:58.233361 6727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:58.237586 6727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:58.237658 6727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:58.237758 6727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:25:58.243381 6727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:39797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:58.244831 6727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:58.245886 6727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:58.255860 6732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:58.255900 6733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:58.255928 6735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:58.256220 6727 server_base.cc:1034] running on GCE node
I20260619 22:25:58.256706 6727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:58.257467 6727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:58.258657 6727 hybrid_clock.cc:648] HybridClock initialized: now 1781907958258644 us; error 37 us; skew 500 ppm
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:58.261514 6727 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:25:58.262486 6727 webserver.cc:460] Webserver started at http://127.5.176.195:40465/ using document root <none> and password file <none>
I20260619 22:25:58.263023 6727 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:58.263080 6727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:58.263247 6727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:58.265300 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "00371cfc06b8481497701beefccfbe4f"
format_stamp: "Formatted at 2026-06-19 22:25:58 on dist-test-slave-bt27"
server_key: "cd20e3b19e94585f7677a2a52f974a55"
server_key_iv: "0c680734158a9b1500a5556434912bdb"
server_key_version: "encryptionkey@0"
I20260619 22:25:58.265813 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "00371cfc06b8481497701beefccfbe4f"
format_stamp: "Formatted at 2026-06-19 22:25:58 on dist-test-slave-bt27"
server_key: "cd20e3b19e94585f7677a2a52f974a55"
server_key_iv: "0c680734158a9b1500a5556434912bdb"
server_key_version: "encryptionkey@0"
I20260619 22:25:58.269774 6727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:25:58.272372 6742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:58.273664 6727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 22:25:58.273761 6727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "00371cfc06b8481497701beefccfbe4f"
format_stamp: "Formatted at 2026-06-19 22:25:58 on dist-test-slave-bt27"
server_key: "cd20e3b19e94585f7677a2a52f974a55"
server_key_iv: "0c680734158a9b1500a5556434912bdb"
server_key_version: "encryptionkey@0"
I20260619 22:25:58.273916 6727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:58.297333 6727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:58.300817 6727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:58.301047 6727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:25:58.301692 6727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:25:58.302636 6727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:25:58.302683 6727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:58.302747 6727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:25:58.302781 6727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:58.311777 6727 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:39725
I20260619 22:25:58.311796 6855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:39725 every 8 connection(s)
I20260619 22:25:58.312687 6727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:25:58.315393 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6727
I20260619 22:25:58.315527 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:25:58.315847 5827 external_mini_cluster.cc:1428] Setting key e70ac99bb4be72755c5d888f05bd607f
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:25:58.327191 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.314655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:34601 (local address 127.5.176.254:39797)
0619 22:25:58.314834 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.314838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.315842 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 22:25:58.317545 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.317549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.317600 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.317691 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:58.319106 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.319642 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.320847 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.321052 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.321157 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.323692 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:58.323695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:58.323697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:58.323722 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:58.325201 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.325729 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.325733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.325735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.325836 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.326182 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.326186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.326187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.326381 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:58.326468 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.326965 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.327065 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 22:25:58.328076 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.315084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:39797 (local address 127.5.176.195:34601)
0619 22:25:58.315622 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.315681 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.317341 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.317998 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.318007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.318469 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:25:58.318978 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.318992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.319787 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.319790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.320717 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.320726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.321196 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.321199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.321348 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.321900 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:25:58.321916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:25:58.323511 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:25:58.325326 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:58.325331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:58.325343 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:58.325626 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:58.325921 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:25:58.325924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:25:58.325925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:25:58.326057 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:25:58.326497 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:25:58.326502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:25:58.326748 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.326922 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.327179 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":107,"threads_started":1}
I20260619 22:25:58.329345 6856 heartbeater.cc:347] Connected to a master server at 127.5.176.254:39797
I20260619 22:25:58.329622 6856 heartbeater.cc:464] Registering TS with master...
I20260619 22:25:58.330258 6856 heartbeater.cc:511] Master 127.5.176.254:39797 requested a full tablet report, sending...
I20260619 22:25:58.331328 6397 ts_manager.cc:194] Registered new tserver with Master: 00371cfc06b8481497701beefccfbe4f (127.5.176.195:39725)
I20260619 22:25:58.331992 6397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:34601
I20260619 22:25:58.341969 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:25:58.353420 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.345569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.5.176.254:39797)
0619 22:25:58.345738 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.345743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.345876 (+ 133us) server_negotiation.cc:407] Connection header received
0619 22:25:58.346049 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.346052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.346101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.346243 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:58.346974 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.347516 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.348351 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.348577 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.348692 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.350078 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:58.350082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:58.350084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:58.350113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:58.351729 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.352155 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.352160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.352162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.352246 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.352523 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.352528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.352529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.352753 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:58.352900 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.353144 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.353259 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 22:25:58.358474 6397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:25:58.361668 6397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:25:58.377373 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.371417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.5.176.194:45709)
0619 22:25:58.371814 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.371827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.371852 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:25:58.372016 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.372024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.372263 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.372411 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:25:58.372966 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.374215 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.376033 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.376667 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.376803 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.376960 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.377089 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.377181 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260619 22:25:58.377424 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.371014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:45709 (local address 127.0.0.1:34952)
0619 22:25:58.371648 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.371696 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.371842 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.372600 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.372609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.372645 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:25:58.372825 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.372836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.374423 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.374426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.375882 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.375893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.376802 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.376805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.376907 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.376941 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.377059 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.377180 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":9,"spinlock_wait_cycles":8192,"thread_start_us":93,"threads_started":1}
I20260619 22:25:58.377687 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.370757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39509 (local address 127.0.0.1:34824)
0619 22:25:58.371641 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.371695 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.371844 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.372607 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.372610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.372649 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:25:58.372838 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.372845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.374377 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.374381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.375767 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.375778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.377087 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.377091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.377374 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.377407 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.377473 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.377544 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":257,"spinlock_wait_cycles":19840,"thread_start_us":78,"threads_started":1}
I20260619 22:25:58.377915 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.370986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.5.176.195:39725)
0619 22:25:58.371435 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.371449 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.372170 (+ 721us) server_negotiation.cc:407] Connection header received
0619 22:25:58.372377 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.372382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.372555 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.372671 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:25:58.373067 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.373881 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.375894 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.376752 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.376877 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.377020 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.377509 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.377603 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":136,"threads_started":1}
I20260619 22:25:58.378105 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.370919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.5.176.193:39509)
0619 22:25:58.372138 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.372155 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.372176 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:25:58.372247 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.372254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.372452 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.372633 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:25:58.372993 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.374199 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.376070 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.376815 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.376904 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.377024 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.377848 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.377965 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":130,"threads_started":1}
I20260619 22:25:58.378283 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.370757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:39725 (local address 127.0.0.1:34774)
0619 22:25:58.372083 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.372095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.372233 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.372850 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.372853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.372874 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:25:58.372956 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.372961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.374010 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.374015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.375775 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.375780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.377087 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.377091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.377328 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.377352 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.378045 (+ 693us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.378125 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":770,"thread_start_us":86,"threads_started":1}
I20260619 22:25:58.381238 6522 tablet_service.cc:1467] Processing CreateTablet for tablet e33a6cc5094e40098b6c55656a0aa6d4 (DEFAULT_TABLE table=test-table [id=50719d300078411c8eede5e83082ecf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:25:58.381276 6656 tablet_service.cc:1467] Processing CreateTablet for tablet e33a6cc5094e40098b6c55656a0aa6d4 (DEFAULT_TABLE table=test-table [id=50719d300078411c8eede5e83082ecf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:25:58.381276 6790 tablet_service.cc:1467] Processing CreateTablet for tablet e33a6cc5094e40098b6c55656a0aa6d4 (DEFAULT_TABLE table=test-table [id=50719d300078411c8eede5e83082ecf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:25:58.382316 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33a6cc5094e40098b6c55656a0aa6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:25:58.382323 6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33a6cc5094e40098b6c55656a0aa6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:25:58.382448 6790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33a6cc5094e40098b6c55656a0aa6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:25:58.389005 6873 tablet_bootstrap.cc:492] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: Bootstrap starting.
I20260619 22:25:58.390645 6875 tablet_bootstrap.cc:492] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: Bootstrap starting.
I20260619 22:25:58.391206 6874 tablet_bootstrap.cc:492] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: Bootstrap starting.
I20260619 22:25:58.391439 6873 tablet_bootstrap.cc:654] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: Neither blocks nor log segments found. Creating new log.
I20260619 22:25:58.392179 6873 log.cc:824] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:25:58.393226 6875 tablet_bootstrap.cc:654] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: Neither blocks nor log segments found. Creating new log.
I20260619 22:25:58.393744 6874 tablet_bootstrap.cc:654] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: Neither blocks nor log segments found. Creating new log.
I20260619 22:25:58.394083 6873 tablet_bootstrap.cc:492] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: No bootstrap required, opened a new log
I20260619 22:25:58.394168 6875 log.cc:824] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:25:58.394353 6873 ts_tablet_manager.cc:1397] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:25:58.394568 6874 log.cc:824] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: Log is configured to *not* fsync() on all Append() calls
I20260619 22:25:58.396130 6875 tablet_bootstrap.cc:492] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: No bootstrap required, opened a new log
I20260619 22:25:58.396378 6875 ts_tablet_manager.cc:1397] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:25:58.396430 6874 tablet_bootstrap.cc:492] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: No bootstrap required, opened a new log
I20260619 22:25:58.396577 6874 ts_tablet_manager.cc:1397] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:25:58.398272 6873 raft_consensus.cc:348] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.398557 6873 raft_consensus.cc:374] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:25:58.398613 6873 raft_consensus.cc:729] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcf95d89b984c47880d89fa3dda844f, State: Initialized, Role: FOLLOWER
I20260619 22:25:58.399084 6873 consensus_queue.cc:260] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.399747 6588 heartbeater.cc:503] Master 127.5.176.254:39797 was elected leader, sending a full tablet report...
I20260619 22:25:58.400748 6875 raft_consensus.cc:348] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.400758 6874 raft_consensus.cc:348] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.401017 6874 raft_consensus.cc:374] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:25:58.401017 6875 raft_consensus.cc:374] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:25:58.401067 6875 raft_consensus.cc:729] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00371cfc06b8481497701beefccfbe4f, State: Initialized, Role: FOLLOWER
I20260619 22:25:58.401073 6874 raft_consensus.cc:729] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f6ba2cf9bb4622b386341c67982d54, State: Initialized, Role: FOLLOWER
I20260619 22:25:58.401155 6873 ts_tablet_manager.cc:1428] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:25:58.401525 6875 consensus_queue.cc:260] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.401525 6874 consensus_queue.cc:260] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.402509 6874 ts_tablet_manager.cc:1428] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:25:58.402491 6856 heartbeater.cc:503] Master 127.5.176.254:39797 was elected leader, sending a full tablet report...
I20260619 22:25:58.402509 6875 ts_tablet_manager.cc:1428] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260619 22:25:58.403609 6722 heartbeater.cc:503] Master 127.5.176.254:39797 was elected leader, sending a full tablet report...
I20260619 22:25:58.511178 6881 raft_consensus.cc:482] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:25:58.511488 6881 raft_consensus.cc:504] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.512746 6881 leader_election.cc:288] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afcf95d89b984c47880d89fa3dda844f (127.5.176.193:39509), 00371cfc06b8481497701beefccfbe4f (127.5.176.195:39725)
I20260619 22:25:58.516604 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.513274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:42883 (local address 127.5.176.193:39509)
0619 22:25:58.513443 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.513449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.513466 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:25:58.513522 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.513529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.513571 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.513668 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:25:58.513959 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.514546 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.515603 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.516018 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.516154 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.516209 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.516398 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.516480 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:25:58.516604 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.513006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39509 (local address 127.5.176.194:42883)
0619 22:25:58.513115 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.513132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.513236 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.513684 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.513687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.513707 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:25:58.513833 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.513839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.514663 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.514668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.515464 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.515474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.516149 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.516153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.516260 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.516282 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.516382 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.516459 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 22:25:58.517000 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.513006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:39725 (local address 127.5.176.194:39129)
0619 22:25:58.513735 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 22:25:58.513750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:25:58.513856 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:25:58.514172 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:25:58.514175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:25:58.514191 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:25:58.514303 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.514309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.515067 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.515073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.515868 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:25:58.515877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.516525 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.516529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:25:58.516624 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.516645 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:25:58.516749 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 22:25:58.516842 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":304,"thread_start_us":152,"threads_started":1}
I20260619 22:25:58.517088 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.513106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:39129 (local address 127.5.176.195:39725)
0619 22:25:58.513249 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.513253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.513827 (+ 574us) server_negotiation.cc:407] Connection header received
0619 22:25:58.513994 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.513997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.514049 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.514136 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:25:58.514429 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.514948 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.515999 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.516404 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.516534 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.516582 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.516889 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.516964 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:25:58.517403 6542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33a6cc5094e40098b6c55656a0aa6d4" candidate_uuid: "e1f6ba2cf9bb4622b386341c67982d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afcf95d89b984c47880d89fa3dda844f" is_pre_election: true
I20260619 22:25:58.517652 6810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33a6cc5094e40098b6c55656a0aa6d4" candidate_uuid: "e1f6ba2cf9bb4622b386341c67982d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00371cfc06b8481497701beefccfbe4f" is_pre_election: true
I20260619 22:25:58.517755 6542 raft_consensus.cc:2454] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f6ba2cf9bb4622b386341c67982d54 in term 0.
I20260619 22:25:58.517956 6810 raft_consensus.cc:2454] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f6ba2cf9bb4622b386341c67982d54 in term 0.
I20260619 22:25:58.518342 6612 leader_election.cc:302] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00371cfc06b8481497701beefccfbe4f, afcf95d89b984c47880d89fa3dda844f, e1f6ba2cf9bb4622b386341c67982d54; no voters:
I20260619 22:25:58.518625 6881 raft_consensus.cc:2781] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:25:58.518711 6881 raft_consensus.cc:482] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:25:58.518754 6881 raft_consensus.cc:3037] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:25:58.519819 6881 raft_consensus.cc:504] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.520226 6881 leader_election.cc:288] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [CANDIDATE]: Term 1 election: Requested vote from peers afcf95d89b984c47880d89fa3dda844f (127.5.176.193:39509), 00371cfc06b8481497701beefccfbe4f (127.5.176.195:39725)
I20260619 22:25:58.520614 6542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33a6cc5094e40098b6c55656a0aa6d4" candidate_uuid: "e1f6ba2cf9bb4622b386341c67982d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afcf95d89b984c47880d89fa3dda844f"
I20260619 22:25:58.520670 6810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33a6cc5094e40098b6c55656a0aa6d4" candidate_uuid: "e1f6ba2cf9bb4622b386341c67982d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00371cfc06b8481497701beefccfbe4f"
I20260619 22:25:58.520741 6542 raft_consensus.cc:3037] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:25:58.520833 6810 raft_consensus.cc:3037] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:25:58.521824 6542 raft_consensus.cc:2454] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f6ba2cf9bb4622b386341c67982d54 in term 1.
I20260619 22:25:58.521936 6810 raft_consensus.cc:2454] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f6ba2cf9bb4622b386341c67982d54 in term 1.
I20260619 22:25:58.522194 6612 leader_election.cc:302] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afcf95d89b984c47880d89fa3dda844f, e1f6ba2cf9bb4622b386341c67982d54; no voters:
I20260619 22:25:58.522356 6881 raft_consensus.cc:2781] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:25:58.522588 6881 raft_consensus.cc:686] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [term 1 LEADER]: Becoming Leader. State: Replica: e1f6ba2cf9bb4622b386341c67982d54, State: Running, Role: LEADER
I20260619 22:25:58.522929 6881 consensus_queue.cc:237] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } }
I20260619 22:25:58.526377 6396 catalog_manager.cc:5495] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1f6ba2cf9bb4622b386341c67982d54 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "e1f6ba2cf9bb4622b386341c67982d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afcf95d89b984c47880d89fa3dda844f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00371cfc06b8481497701beefccfbe4f" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f6ba2cf9bb4622b386341c67982d54" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 45709 } health_report { overall_health: HEALTHY } } }
W20260619 22:25:58.563701 6857 tablet.cc:2369] T e33a6cc5094e40098b6c55656a0aa6d4 P 00371cfc06b8481497701beefccfbe4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:25:58.575372 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.564797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.5.176.193:39509)
0619 22:25:58.564998 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.565005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.565247 (+ 242us) server_negotiation.cc:407] Connection header received
0619 22:25:58.565388 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.565390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.565434 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.565559 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:58.566331 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.567197 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.568085 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.568308 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.568399 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.571344 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:58.571364 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:58.571379 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:58.571407 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:58.573290 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.573800 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.573807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.573809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.573904 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.574238 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.574241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.574243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.574512 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:58.574771 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.575041 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.575178 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 22:25:58.582245 6723 tablet.cc:2369] T e33a6cc5094e40098b6c55656a0aa6d4 P e1f6ba2cf9bb4622b386341c67982d54: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:25:58.602849 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.593860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.5.176.193:39509)
0619 22:25:58.594065 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.594069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.594228 (+ 159us) server_negotiation.cc:407] Connection header received
0619 22:25:58.594392 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.594394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.594447 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.594518 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:58.595188 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.595704 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.596567 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.596784 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.596860 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.599501 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:58.599505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:58.599507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:58.599531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:58.601050 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.601479 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.601481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.601482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.601554 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.601842 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.601844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.601845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.602029 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:58.602235 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.602543 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.602648 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260619 22:25:58.609047 6589 tablet.cc:2369] T e33a6cc5094e40098b6c55656a0aa6d4 P afcf95d89b984c47880d89fa3dda844f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:25:58.629169 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:25:58.620672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.5.176.193:39509)
0619 22:25:58.620874 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 22:25:58.620879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:25:58.620947 (+ 68us) server_negotiation.cc:407] Connection header received
0619 22:25:58.621065 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:25:58.621068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:25:58.621112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:25:58.621183 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:25:58.621859 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.622438 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.623247 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:25:58.623464 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:25:58.623545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:25:58.626058 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:25:58.626062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:25:58.626064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:25:58.626093 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:25:58.627615 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.628003 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.628005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.628006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.628079 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:25:58.628334 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:25:58.628337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:25:58.628338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:25:58.628518 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:25:58.628631 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:25:58.628829 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 22:25:58.628939 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260619 22:25:58.630656 6502 tablet_service.cc:2908] Rejecting scan request for tablet e33a6cc5094e40098b6c55656a0aa6d4: Uninitialized: clean time has not yet been initialized
I20260619 22:25:58.632867 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6459
I20260619 22:25:58.638963 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6593
I20260619 22:25:58.644740 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6727
I20260619 22:25:58.650343 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6365
2026-06-19T22:25:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2472 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6897](info): set up 2 sockets
Jun 19 22:25:58 dist-test-slave-bt27 krb5kdc[6897](info): commencing operation
krb5kdc: starting...
W20260619 22:25:59.697831 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:25:59 dist-test-slave-bt27 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:25:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:25:59.817515 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:42519
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:25:59.929774 6910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:25:59.930052 6910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:25:59.930096 6910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:25:59.934208 6910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:25:59.934270 6910 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 22:25:59.934288 6910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:25:59.934305 6910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:25:59.934321 6910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:25:59.934336 6910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:25:59.939733 6910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:42519
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:25:59.940788 6910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:25:59.941753 6910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:25:59.950372 6915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:59.950438 6910 server_base.cc:1034] running on GCE node
W20260619 22:25:59.950380 6918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:25:59.950377 6916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:25:59.951136 6910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:25:59.952286 6910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:25:59.953485 6910 hybrid_clock.cc:648] HybridClock initialized: now 1781907959953461 us; error 38 us; skew 500 ppm
Jun 19 22:25:59 dist-test-slave-bt27 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:25:59.956198 6910 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:25:59.957084 6910 webserver.cc:460] Webserver started at http://127.5.176.254:34587/ using document root <none> and password file <none>
I20260619 22:25:59.957674 6910 fs_manager.cc:359] Metadata directory not provided
I20260619 22:25:59.957770 6910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:25:59.958003 6910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:25:59.960158 6910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "7a030265df674202b801c3bd9339d79a"
format_stamp: "Formatted at 2026-06-19 22:25:59 on dist-test-slave-bt27"
server_key: "965dca2406c3284bbc5dead11c5d5ff4"
server_key_iv: "ec4f7596b6643a943611213a91bb86af"
server_key_version: "encryptionkey@0"
I20260619 22:25:59.960754 6910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "7a030265df674202b801c3bd9339d79a"
format_stamp: "Formatted at 2026-06-19 22:25:59 on dist-test-slave-bt27"
server_key: "965dca2406c3284bbc5dead11c5d5ff4"
server_key_iv: "ec4f7596b6643a943611213a91bb86af"
server_key_version: "encryptionkey@0"
I20260619 22:25:59.964618 6910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:25:59.967119 6925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:25:59.968457 6910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:25:59.968699 6910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "7a030265df674202b801c3bd9339d79a"
format_stamp: "Formatted at 2026-06-19 22:25:59 on dist-test-slave-bt27"
server_key: "965dca2406c3284bbc5dead11c5d5ff4"
server_key_iv: "ec4f7596b6643a943611213a91bb86af"
server_key_version: "encryptionkey@0"
I20260619 22:25:59.968825 6910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:25:59.991094 6910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:25:59.994652 6910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:25:59.994904 6910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:00.008358 6910 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:42519
I20260619 22:26:00.008387 6987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:42519 every 8 connection(s)
I20260619 22:26:00.009462 6910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:00.013129 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:00.014550 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 6910
I20260619 22:26:00.014684 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:00.014986 5827 external_mini_cluster.cc:1428] Setting key bc77e00e2ce902619677c0fb367775de
I20260619 22:26:00.021087 6988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: Bootstrap starting.
I20260619 22:26:00.024197 6988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:00.025199 6988 log.cc:824] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:00.027945 6988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: No bootstrap required, opened a new log
I20260619 22:26:00.031850 6988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a030265df674202b801c3bd9339d79a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42519 } }
I20260619 22:26:00.032425 6988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:00.032518 6988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a030265df674202b801c3bd9339d79a, State: Initialized, Role: FOLLOWER
I20260619 22:26:00.033191 6988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a030265df674202b801c3bd9339d79a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42519 } }
I20260619 22:26:00.033380 6988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:00.033499 6988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:00.033654 6988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:00.033793 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.017061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.5.176.254:42519)
0619 22:26:00.017766 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.017783 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.017830 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:26:00.018851 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.018875 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.019556 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.020290 (+ 734us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.021386 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.022720 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.024047 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.024690 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.024830 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.028094 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.028101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.028122 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.028159 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.030005 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.030562 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.030567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.030573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.030694 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.030999 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.031004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.031006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.031556 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.031850 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.032107 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.032515 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":173,"threads_started":1}
I20260619 22:26:00.035099 6988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a030265df674202b801c3bd9339d79a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42519 } }
I20260619 22:26:00.035811 6988 leader_election.cc:302] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a030265df674202b801c3bd9339d79a; no voters:
I20260619 22:26:00.036260 6988 leader_election.cc:288] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:00.036376 6993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:00.036566 6993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [term 1 LEADER]: Becoming Leader. State: Replica: 7a030265df674202b801c3bd9339d79a, State: Running, Role: LEADER
I20260619 22:26:00.036935 6993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a030265df674202b801c3bd9339d79a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42519 } }
I20260619 22:26:00.037984 6988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:00.039121 6994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a030265df674202b801c3bd9339d79a. Latest consensus state: current_term: 1 leader_uuid: "7a030265df674202b801c3bd9339d79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a030265df674202b801c3bd9339d79a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42519 } } }
I20260619 22:26:00.039292 6994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:00.039094 6995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a030265df674202b801c3bd9339d79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a030265df674202b801c3bd9339d79a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42519 } } }
I20260619 22:26:00.039535 6995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:00.039798 7002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:00.043154 7002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:00.050194 7002 catalog_manager.cc:1347] Generated new cluster ID: 37f6ef6e30fe4b41a6b706d37145a227
I20260619 22:26:00.050287 7002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:00.071111 7002 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:00.072300 7002 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:00.078233 7002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: Generated new TSK 0
I20260619 22:26:00.079151 7002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:00.115459 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42519
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:00.237653 7014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:00.238106 7014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:00.238229 7014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:00.242590 7014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:00.242694 7014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:00.242769 7014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:00.242851 7014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:00.248512 7014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:00.249660 7014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:00.250747 7014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:00.260881 7019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:00.261011 7020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:00.261054 7014 server_base.cc:1034] running on GCE node
W20260619 22:26:00.260881 7022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:00.261597 7014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:00.262375 7014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:00.263599 7014 hybrid_clock.cc:648] HybridClock initialized: now 1781907960263552 us; error 52 us; skew 500 ppm
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:00.267127 7014 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:00.268162 7014 webserver.cc:460] Webserver started at http://127.5.176.193:38509/ using document root <none> and password file <none>
I20260619 22:26:00.268775 7014 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:00.268895 7014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:00.269138 7014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:00.271370 7014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "81ec295f201143b3a63cb5596b25cb70"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "3c020e2623b37d55601e8657d766ea76"
server_key_iv: "9a54a53f78938b761600432dd7d2021b"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.272100 7014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "81ec295f201143b3a63cb5596b25cb70"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "3c020e2623b37d55601e8657d766ea76"
server_key_iv: "9a54a53f78938b761600432dd7d2021b"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.276420 7014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:00.278985 7029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:00.280344 7014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:00.280583 7014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "81ec295f201143b3a63cb5596b25cb70"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "3c020e2623b37d55601e8657d766ea76"
server_key_iv: "9a54a53f78938b761600432dd7d2021b"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.280714 7014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:00.298462 7014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:00.301713 7014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:00.301946 7014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:00.303562 7014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:00.303643 7014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:00.303714 7014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:00.303763 7014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:00.314719 7014 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:34545
I20260619 22:26:00.314800 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:34545 every 8 connection(s)
I20260619 22:26:00.315891 7014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:00.317572 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.304349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55403 (local address 127.5.176.254:42519)
0619 22:26:00.304541 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.304548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.305378 (+ 830us) server_negotiation.cc:407] Connection header received
0619 22:26:00.306541 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.306544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.306608 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.306656 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.308055 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.308674 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.309975 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.310343 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.310468 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.313655 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.313659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.313662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.313691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.315473 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.316072 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.316076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.316078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.316163 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.316551 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.316554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.316556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.316788 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.316874 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.317346 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.317454 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:26:00.318467 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.304739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.5.176.193:55403)
0619 22:26:00.305204 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.305254 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.306279 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.306978 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.306984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.307326 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:00.307845 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.307866 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.308807 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.308811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.309845 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.309855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.310515 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.310519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.310728 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.311384 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:00.311406 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:00.313490 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:00.315595 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.315600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.315611 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.315980 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.316262 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.316266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.316268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.316440 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.316894 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:00.316899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:00.317137 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.317299 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.317550 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
I20260619 22:26:00.319772 7145 heartbeater.cc:347] Connected to a master server at 127.5.176.254:42519
I20260619 22:26:00.319927 7145 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:00.320416 7145 heartbeater.cc:511] Master 127.5.176.254:42519 requested a full tablet report, sending...
I20260619 22:26:00.322132 6942 ts_manager.cc:194] Registered new tserver with Master: 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545)
I20260619 22:26:00.323272 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7014
I20260619 22:26:00.323388 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:00.323751 6942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:55403
I20260619 22:26:00.323962 5827 external_mini_cluster.cc:1428] Setting key 1628240c0999577f4a34ac7dfd4cc05c
I20260619 22:26:00.330684 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.322019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:40017 (local address 127.5.176.254:42519)
0619 22:26:00.322176 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.322184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.322310 (+ 126us) server_negotiation.cc:407] Connection header received
0619 22:26:00.322423 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.322426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.322476 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.322560 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.323327 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.324281 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.325521 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.325882 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.326003 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.327340 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.327344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.327346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.327371 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.328790 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.329300 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.329304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.329306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.329388 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.329763 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.329768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.329770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.329964 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.330077 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.330393 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.330510 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:00.330703 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.321914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.5.176.193:40017)
0619 22:26:00.322221 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.322235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.322343 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.322582 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.322584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.322751 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:00.323178 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.323191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.324469 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.324471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.325389 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.325401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.326022 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.326026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.326120 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.326812 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:00.326824 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:00.327170 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:00.328934 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.328948 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.328950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.329183 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.329583 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.329586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.329587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.329667 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.330064 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:00.330067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:00.330216 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.330338 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.330502 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:00.353030 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42519
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:00.471267 7154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:00.471592 7154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:00.471666 7154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:00.476538 7154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:00.476681 7154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:00.476768 7154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:00.476854 7154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:00.482882 7154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:00.484144 7154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:00.485281 7154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:00.495054 7159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:00.495054 7162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:00.495097 7160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:00.495467 7154 server_base.cc:1034] running on GCE node
I20260619 22:26:00.495929 7154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:00.496578 7154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:00.497736 7154 hybrid_clock.cc:648] HybridClock initialized: now 1781907960497721 us; error 27 us; skew 500 ppm
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:00.500530 7154 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:00.501492 7154 webserver.cc:460] Webserver started at http://127.5.176.194:44701/ using document root <none> and password file <none>
I20260619 22:26:00.502069 7154 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:00.502189 7154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:00.502429 7154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:00.504518 7154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "c25a28e21460405480080c74c8c9a6b9"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "675ff667ba8358b099b6cfa500087a69"
server_key_iv: "00d265cac289a1415059b434da0fd2c9"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.505088 7154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "c25a28e21460405480080c74c8c9a6b9"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "675ff667ba8358b099b6cfa500087a69"
server_key_iv: "00d265cac289a1415059b434da0fd2c9"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.508828 7154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 22:26:00.511740 7169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:00.513507 7154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:26:00.513760 7154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "c25a28e21460405480080c74c8c9a6b9"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "675ff667ba8358b099b6cfa500087a69"
server_key_iv: "00d265cac289a1415059b434da0fd2c9"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.513911 7154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:00.530936 7154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:00.534567 7154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:00.534785 7154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:00.536398 7154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:00.536482 7154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:00.536557 7154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:00.536607 7154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:00.547895 7154 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:44053
I20260619 22:26:00.547914 7284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:44053 every 8 connection(s)
I20260619 22:26:00.548816 7154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:00.549980 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7154
I20260619 22:26:00.550086 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:00.550408 5827 external_mini_cluster.cc:1428] Setting key 4d75dc4d90a9729ab39ce58f2a225043
I20260619 22:26:00.550621 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.537340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:55913 (local address 127.5.176.254:42519)
0619 22:26:00.537535 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.537541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.538494 (+ 953us) server_negotiation.cc:407] Connection header received
0619 22:26:00.539783 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.539790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.539874 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.539932 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.541534 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.542356 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.543559 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.543882 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.544018 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.546687 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.546690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.546692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.546718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.548285 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.548880 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.548883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.548885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.549009 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.549394 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.549398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.549399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.549635 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.549755 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.550319 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.550458 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:26:00.551476 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.537762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.5.176.194:55913)
0619 22:26:00.538303 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.538358 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.539504 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.540383 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.540392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.540817 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:00.541320 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.541334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.542490 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.542494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.543399 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.543409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.544074 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.544078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.544255 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.544921 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:00.544947 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:00.546493 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:00.548408 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.548417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.548435 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.548760 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.549104 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.549107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.549109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.549275 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.549763 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:00.549768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:00.549999 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.550264 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.550606 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":160,"threads_started":1}
I20260619 22:26:00.552868 7285 heartbeater.cc:347] Connected to a master server at 127.5.176.254:42519
I20260619 22:26:00.553022 7285 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:00.553630 7285 heartbeater.cc:511] Master 127.5.176.254:42519 requested a full tablet report, sending...
I20260619 22:26:00.554688 6942 ts_manager.cc:194] Registered new tserver with Master: c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053)
I20260619 22:26:00.555531 6942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:55913
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:26:00.563202 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.555104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.5.176.194:36699)
0619 22:26:00.555452 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.555464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.555579 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.555854 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.555857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.556093 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:00.556213 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.556220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.557249 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.557253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.558020 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.558029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.558549 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.558553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.558645 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.559212 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:00.559232 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:00.559559 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:00.561400 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.561404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.561406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.561623 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.561918 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.561921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.561922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.562010 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.562525 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:00.562527 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:00.562615 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.562813 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.562963 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260619 22:26:00.563210 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.555151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:36699 (local address 127.5.176.254:42519)
0619 22:26:00.555290 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.555294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.555538 (+ 244us) server_negotiation.cc:407] Connection header received
0619 22:26:00.555694 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.555697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.555744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.555833 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.556347 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.557089 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.558169 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.558410 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.558542 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.559705 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.559709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.559712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.559736 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.561272 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.561716 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.561718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.561719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.561793 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.562099 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.562101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.562102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.562393 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.562487 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.562802 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.562955 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:00.581543 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42519
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:00.699266 7294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:00.699568 7294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:00.699616 7294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:00.703876 7294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:00.703944 7294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:00.703987 7294 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:00.704020 7294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:00.709625 7294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:00.710811 7294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:00.711833 7294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:00.722237 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:00.722231 7299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:00.722237 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:00.722630 7294 server_base.cc:1034] running on GCE node
I20260619 22:26:00.723135 7294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:00.723853 7294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:00.725057 7294 hybrid_clock.cc:648] HybridClock initialized: now 1781907960725039 us; error 29 us; skew 500 ppm
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:00.728063 7294 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:00.728922 7294 webserver.cc:460] Webserver started at http://127.5.176.195:44037/ using document root <none> and password file <none>
I20260619 22:26:00.729446 7294 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:00.729503 7294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:00.729667 7294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:00.731777 7294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "b083c0e812df42c9bfcce2918aaaf55b"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "afcc099d9628b21bbd341bc034dbf93b"
server_key_iv: "53dd57c0c815aa24a82ab0ceba81a89b"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.732293 7294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "b083c0e812df42c9bfcce2918aaaf55b"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "afcc099d9628b21bbd341bc034dbf93b"
server_key_iv: "53dd57c0c815aa24a82ab0ceba81a89b"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.736169 7294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:00.738883 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:00.740326 7294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:00.740447 7294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "b083c0e812df42c9bfcce2918aaaf55b"
format_stamp: "Formatted at 2026-06-19 22:26:00 on dist-test-slave-bt27"
server_key: "afcc099d9628b21bbd341bc034dbf93b"
server_key_iv: "53dd57c0c815aa24a82ab0ceba81a89b"
server_key_version: "encryptionkey@0"
I20260619 22:26:00.740556 7294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:00.757712 7294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:00.761135 7294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:00.761370 7294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:00.763019 7294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:00.763100 7294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:00.763170 7294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:00.763219 7294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:00.774201 7294 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:44687
I20260619 22:26:00.774636 7424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:44687 every 8 connection(s)
I20260619 22:26:00.775259 7294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:00.777624 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.763776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:60269 (local address 127.5.176.254:42519)
0619 22:26:00.763951 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.763957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.765417 (+ 1460us) server_negotiation.cc:407] Connection header received
0619 22:26:00.766883 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.766888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.766952 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.767077 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.768330 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.768941 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.770333 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.770561 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.770704 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.773772 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.773779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.773784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.773822 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.775601 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.776229 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.776233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.776234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.776330 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.776656 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.776658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.776659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.776847 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.776930 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.777384 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.777503 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:26:00.778642 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.764674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.5.176.195:60269)
0619 22:26:00.765230 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.765299 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.766632 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.767311 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.767318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.767684 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:00.768175 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.768188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.769096 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.769100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.770186 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.770202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.770760 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.770764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.771007 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.771649 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:00.771681 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:00.773574 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:00.775726 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.775732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.775748 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.776113 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.776435 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.776438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.776439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.776571 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.776935 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:00.776940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:00.777181 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.777341 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.777597 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
I20260619 22:26:00.778820 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7294
I20260619 22:26:00.778940 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:00.779219 5827 external_mini_cluster.cc:1428] Setting key 85e623b7bc029831971e31ea1ef1d311
I20260619 22:26:00.780030 7425 heartbeater.cc:347] Connected to a master server at 127.5.176.254:42519
I20260619 22:26:00.780177 7425 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:00.780776 7425 heartbeater.cc:511] Master 127.5.176.254:42519 requested a full tablet report, sending...
I20260619 22:26:00.782018 6942 ts_manager.cc:194] Registered new tserver with Master: b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195:44687)
I20260619 22:26:00.782840 6942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:60269
I20260619 22:26:00.789585 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.782600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:35957 (local address 127.5.176.254:42519)
0619 22:26:00.782700 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.782705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.782872 (+ 167us) server_negotiation.cc:407] Connection header received
0619 22:26:00.782996 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.782999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.783043 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.783128 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.783618 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.784375 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.785324 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.785528 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.785606 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.786676 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.786679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.786681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.786701 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.788125 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.788550 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.788553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.788555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.788639 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.788903 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.788906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.788907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.789089 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.789169 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.789373 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.789470 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:26:00.789600 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.782526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.5.176.195:35957)
0619 22:26:00.782786 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.782814 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.782911 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.783165 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.783178 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.783389 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:00.783505 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.783510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.784501 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.784504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.785214 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.785223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.785622 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.785625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.785692 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.786176 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:00.786196 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:00.786542 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:00.788226 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.788228 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.788230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.788422 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.788728 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:00.788730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:00.788732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:00.788808 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:00.789174 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:00.789176 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:00.789249 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.789338 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.789443 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260619 22:26:00.793419 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:26:00 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:00.821157 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.811593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.5.176.254:42519)
0619 22:26:00.811740 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.811744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.811778 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:26:00.811819 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.811821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.811862 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.811930 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:00.812731 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.813290 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.814237 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.814470 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.814561 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.817554 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:00.817558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:00.817560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:00.817591 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:00.819305 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.819735 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.819737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.819739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.819865 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:00.820173 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:00.820175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:00.820176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:00.820366 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:00.820474 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.820799 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.821000 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:26:00.825035 6942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:00.828317 6942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:00.843631 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.838267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.5.176.195:44687)
0619 22:26:00.838687 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.838703 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.838728 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:26:00.838819 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.838828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.839039 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.839192 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:00.839581 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.840448 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.842303 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.842861 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.843013 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.843193 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.843313 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.843408 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":111,"threads_started":1}
I20260619 22:26:00.843611 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.837143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:44053 (local address 127.0.0.1:35254)
0619 22:26:00.837667 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.837716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.837848 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.839098 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.839101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.839113 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:00.839252 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.839259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.840862 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.840865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.842000 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.842009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.843144 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.843148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.843250 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.843272 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.843336 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.843429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":39,"thread_start_us":64,"threads_started":1}
I20260619 22:26:00.843622 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.837305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.5.176.194:44053)
0619 22:26:00.837893 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.837909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.837981 (+ 72us) server_negotiation.cc:407] Connection header received
0619 22:26:00.838065 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.838070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.838348 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.839387 (+ 1039us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:00.839518 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.840681 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.842165 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.842998 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.843192 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.843359 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.843406 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.843497 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":104,"threads_started":1}
I20260619 22:26:00.844081 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.837302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34545 (local address 127.0.0.1:47582)
0619 22:26:00.838124 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.838184 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.838271 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.838994 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.839004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.839031 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:00.839291 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.839301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.840781 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.840784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.842076 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.842087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.843144 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.843148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.843749 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.843783 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.843857 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.843931 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":98,"thread_start_us":106,"threads_started":1}
I20260619 22:26:00.843592 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.837869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:44687 (local address 127.0.0.1:42034)
0619 22:26:00.838499 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:00.838515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:00.838601 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:00.839172 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:00.839175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:00.839194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:00.839465 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.839469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.840557 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.840563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.842210 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:00.842218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.843015 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.843019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:00.843123 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.843159 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:26:00.843284 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 22:26:00.843412 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":45,"thread_start_us":89,"threads_started":1}
I20260619 22:26:00.844792 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:00.837781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.5.176.193:34545)
0619 22:26:00.838525 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:00.838534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:00.838557 (+ 23us) server_negotiation.cc:407] Connection header received
0619 22:26:00.838631 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:00.838639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:00.838824 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:00.839311 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:00.839460 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.840658 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.842230 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:00.842998 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:00.844051 (+ 1053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:00.844301 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:00.844532 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 22:26:00.844652 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":63,"threads_started":1}
I20260619 22:26:00.847075 7359 tablet_service.cc:1467] Processing CreateTablet for tablet 18c1a98d2b0448cda902d2ac0cde57d3 (DEFAULT_TABLE table=test-table [id=c9cfeda1e41a40ac959fb0ef77bd2923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:00.847092 7219 tablet_service.cc:1467] Processing CreateTablet for tablet 18c1a98d2b0448cda902d2ac0cde57d3 (DEFAULT_TABLE table=test-table [id=c9cfeda1e41a40ac959fb0ef77bd2923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:00.847167 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 18c1a98d2b0448cda902d2ac0cde57d3 (DEFAULT_TABLE table=test-table [id=c9cfeda1e41a40ac959fb0ef77bd2923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:00.848064 7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c1a98d2b0448cda902d2ac0cde57d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:00.848070 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c1a98d2b0448cda902d2ac0cde57d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:00.848070 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c1a98d2b0448cda902d2ac0cde57d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:00.854980 7448 tablet_bootstrap.cc:492] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Bootstrap starting.
I20260619 22:26:00.856573 7450 tablet_bootstrap.cc:492] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Bootstrap starting.
I20260619 22:26:00.857419 7448 tablet_bootstrap.cc:654] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:00.858355 7448 log.cc:824] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:00.858433 7449 tablet_bootstrap.cc:492] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Bootstrap starting.
I20260619 22:26:00.858840 7450 tablet_bootstrap.cc:654] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:00.859730 7450 log.cc:824] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:00.860566 7448 tablet_bootstrap.cc:492] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: No bootstrap required, opened a new log
I20260619 22:26:00.860566 7449 tablet_bootstrap.cc:654] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:00.860770 7448 ts_tablet_manager.cc:1397] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:00.861374 7449 log.cc:824] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:00.861591 7450 tablet_bootstrap.cc:492] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: No bootstrap required, opened a new log
I20260619 22:26:00.861793 7450 ts_tablet_manager.cc:1397] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:26:00.863811 7449 tablet_bootstrap.cc:492] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: No bootstrap required, opened a new log
I20260619 22:26:00.863994 7449 ts_tablet_manager.cc:1397] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:00.864863 7448 raft_consensus.cc:348] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:00.865072 7448 raft_consensus.cc:374] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:00.865104 7448 raft_consensus.cc:729] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b083c0e812df42c9bfcce2918aaaf55b, State: Initialized, Role: FOLLOWER
I20260619 22:26:00.865530 7448 consensus_queue.cc:260] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:00.865669 7450 raft_consensus.cc:348] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:00.865902 7450 raft_consensus.cc:374] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:00.865947 7450 raft_consensus.cc:729] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25a28e21460405480080c74c8c9a6b9, State: Initialized, Role: FOLLOWER
I20260619 22:26:00.866160 7425 heartbeater.cc:503] Master 127.5.176.254:42519 was elected leader, sending a full tablet report...
I20260619 22:26:00.866382 7450 consensus_queue.cc:260] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:00.866532 7448 ts_tablet_manager.cc:1428] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:00.867540 7450 ts_tablet_manager.cc:1428] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:26:00.867889 7285 heartbeater.cc:503] Master 127.5.176.254:42519 was elected leader, sending a full tablet report...
I20260619 22:26:00.871481 7449 raft_consensus.cc:348] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:00.871723 7449 raft_consensus.cc:374] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:00.871798 7449 raft_consensus.cc:729] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ec295f201143b3a63cb5596b25cb70, State: Initialized, Role: FOLLOWER
I20260619 22:26:00.872256 7449 consensus_queue.cc:260] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:00.872915 7145 heartbeater.cc:503] Master 127.5.176.254:42519 was elected leader, sending a full tablet report...
I20260619 22:26:00.873246 7449 ts_tablet_manager.cc:1428] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260619 22:26:01.025072 7454 raft_consensus.cc:482] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:01.025321 7454 raft_consensus.cc:504] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
W20260619 22:26:01.026566 7426 tablet.cc:2369] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:01.026597 7454 leader_election.cc:288] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053), 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545)
I20260619 22:26:01.031383 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.027097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:51023 (local address 127.5.176.193:34545)
0619 22:26:01.027231 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.027237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.027253 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:01.027310 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.027312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.027363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.027444 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:01.027843 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.028673 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.030016 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.030756 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.030882 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.030967 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.031161 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.031236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:26:01.031383 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.026888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34545 (local address 127.5.176.195:51023)
0619 22:26:01.027097 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.027113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.027207 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.027475 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.027477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.027495 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:01.027659 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.027666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.028802 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.028806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.029846 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.029863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.030882 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.030886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.031028 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.031046 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.031144 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.031230 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 22:26:01.032150 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.027008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:53385 (local address 127.5.176.194:44053)
0619 22:26:01.027140 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.027145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.027594 (+ 449us) server_negotiation.cc:407] Connection header received
0619 22:26:01.027658 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.027661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.027718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.027817 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:01.028593 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.029367 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.030816 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.031252 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.031412 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.031461 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.031982 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.032057 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:01.032150 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.026888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:44053 (local address 127.5.176.195:53385)
0619 22:26:01.027471 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.027485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.027590 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.027839 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.027843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.027864 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:01.028437 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.028446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.029514 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.029517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.030611 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.030622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.031419 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.031423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.031788 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.031818 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.031934 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.032006 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":230,"thread_start_us":135,"threads_started":1}
I20260619 22:26:01.032207 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c1a98d2b0448cda902d2ac0cde57d3" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ec295f201143b3a63cb5596b25cb70" is_pre_election: true
I20260619 22:26:01.032550 7099 raft_consensus.cc:2454] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 0.
I20260619 22:26:01.032894 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c1a98d2b0448cda902d2ac0cde57d3" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25a28e21460405480080c74c8c9a6b9" is_pre_election: true
I20260619 22:26:01.033149 7311 leader_election.cc:302] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ec295f201143b3a63cb5596b25cb70, b083c0e812df42c9bfcce2918aaaf55b; no voters:
I20260619 22:26:01.033218 7239 raft_consensus.cc:2454] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 0.
I20260619 22:26:01.033458 7454 raft_consensus.cc:2781] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:01.033568 7454 raft_consensus.cc:482] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:01.033627 7454 raft_consensus.cc:3037] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.034694 7454 raft_consensus.cc:504] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.035107 7454 leader_election.cc:288] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 election: Requested vote from peers c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053), 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545)
I20260619 22:26:01.035437 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c1a98d2b0448cda902d2ac0cde57d3" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25a28e21460405480080c74c8c9a6b9"
I20260619 22:26:01.035497 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c1a98d2b0448cda902d2ac0cde57d3" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ec295f201143b3a63cb5596b25cb70"
I20260619 22:26:01.035576 7239 raft_consensus.cc:3037] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.035642 7099 raft_consensus.cc:3037] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.036684 7099 raft_consensus.cc:2454] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 1.
I20260619 22:26:01.036693 7239 raft_consensus.cc:2454] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 1.
I20260619 22:26:01.037014 7312 leader_election.cc:302] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b083c0e812df42c9bfcce2918aaaf55b, c25a28e21460405480080c74c8c9a6b9; no voters:
I20260619 22:26:01.037217 7454 raft_consensus.cc:2781] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:01.037400 7454 raft_consensus.cc:686] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 1 LEADER]: Becoming Leader. State: Replica: b083c0e812df42c9bfcce2918aaaf55b, State: Running, Role: LEADER
I20260619 22:26:01.037777 7454 consensus_queue.cc:237] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.041142 6942 catalog_manager.cc:5495] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b reported cstate change: term changed from 0 to 1, leader changed from <none> to b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195). New cstate: current_term: 1 leader_uuid: "b083c0e812df42c9bfcce2918aaaf55b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } health_report { overall_health: HEALTHY } } }
W20260619 22:26:01.050225 7286 tablet.cc:2369] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:01.067400 7146 tablet.cc:2369] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:01.085495 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.081448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.5.176.195:44687)
0619 22:26:01.081573 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.081576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.081589 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:01.081801 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.081804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.081858 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.081950 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:01.082248 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.082758 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.083693 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.083951 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.084091 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.084447 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.085054 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.085144 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.085312 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.085377 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:01.092154 7099 raft_consensus.cc:1261] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Refusing update from remote peer b083c0e812df42c9bfcce2918aaaf55b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:01.092185 7239 raft_consensus.cc:1261] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Refusing update from remote peer b083c0e812df42c9bfcce2918aaaf55b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:01.092912 7454 consensus_queue.cc:1035] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [LEADER]: Connected to new peer: Peer: permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:01.093178 7458 consensus_queue.cc:1035] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:01.099862 7461 mvcc.cc:205] Tried to move back new op lower bound from 7298695008625958912 to 7298695008413765632. Current Snapshot: MvccSnapshot[applied={T|T < 7298695008625958912}]
I20260619 22:26:01.102624 7463 mvcc.cc:205] Tried to move back new op lower bound from 7298695008625958912 to 7298695008413765632. Current Snapshot: MvccSnapshot[applied={T|T < 7298695008625958912 or (T in {7298695008625958912})}]
I20260619 22:26:01.103181 7462 mvcc.cc:205] Tried to move back new op lower bound from 7298695008625958912 to 7298695008413765632. Current Snapshot: MvccSnapshot[applied={T|T < 7298695008625958912}]
I20260619 22:26:01.113842 6937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:01.114028 6937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:01.116812 6937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: Sending DeleteTablet for 3 replicas of tablet 18c1a98d2b0448cda902d2ac0cde57d3
I20260619 22:26:01.117798 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c1a98d2b0448cda902d2ac0cde57d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:01 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:47582
I20260619 22:26:01.117848 7219 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c1a98d2b0448cda902d2ac0cde57d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:01 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:35254
I20260619 22:26:01.118628 7470 tablet_replica.cc:330] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: stopping tablet replica
I20260619 22:26:01.118676 7359 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c1a98d2b0448cda902d2ac0cde57d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:01 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:42034
I20260619 22:26:01.118836 6937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 22:26:01.119206 7471 tablet_replica.cc:330] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: stopping tablet replica
W20260619 22:26:01.119273 6937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:01.119334 7470 raft_consensus.cc:2229] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:01.119498 7471 raft_consensus.cc:2229] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:01.119828 7470 raft_consensus.cc:2258] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:01.119833 7471 raft_consensus.cc:2258] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:01.121172 7472 tablet_replica.cc:330] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: stopping tablet replica
I20260619 22:26:01.121469 7472 raft_consensus.cc:2229] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:01.121737 7471 ts_tablet_manager.cc:1905] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:01.121815 7472 raft_consensus.cc:2258] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:01.123260 7472 ts_tablet_manager.cc:1905] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:01.124804 7471 ts_tablet_manager.cc:1918] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:01.124913 7471 log.cc:1196] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/18c1a98d2b0448cda902d2ac0cde57d3
I20260619 22:26:01.125249 7471 ts_tablet_manager.cc:1939] T 18c1a98d2b0448cda902d2ac0cde57d3 P 81ec295f201143b3a63cb5596b25cb70: Deleting consensus metadata
I20260619 22:26:01.125957 7472 ts_tablet_manager.cc:1918] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:01.126041 7472 log.cc:1196] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/18c1a98d2b0448cda902d2ac0cde57d3
I20260619 22:26:01.126390 7359 tablet_service.cc:1467] Processing CreateTablet for tablet 13a80a229cc54cd9abbc53c4aeb01a89 (DEFAULT_TABLE table=test-table [id=cefcfcd152f647959bcad3aa0db9c078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:01.126410 7219 tablet_service.cc:1467] Processing CreateTablet for tablet 13a80a229cc54cd9abbc53c4aeb01a89 (DEFAULT_TABLE table=test-table [id=cefcfcd152f647959bcad3aa0db9c078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:01.126652 7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a80a229cc54cd9abbc53c4aeb01a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:01.126683 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a80a229cc54cd9abbc53c4aeb01a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:01.128593 7450 tablet_bootstrap.cc:492] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Bootstrap starting.
I20260619 22:26:01.128952 7448 tablet_bootstrap.cc:492] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Bootstrap starting.
I20260619 22:26:01.126380 7472 ts_tablet_manager.cc:1939] T 18c1a98d2b0448cda902d2ac0cde57d3 P b083c0e812df42c9bfcce2918aaaf55b: Deleting consensus metadata
I20260619 22:26:01.129535 6927 catalog_manager.cc:4841] TS 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545): tablet 18c1a98d2b0448cda902d2ac0cde57d3 (table test-table [id=c9cfeda1e41a40ac959fb0ef77bd2923]) successfully deleted
I20260619 22:26:01.129735 7450 tablet_bootstrap.cc:654] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:01.129870 7448 tablet_bootstrap.cc:654] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:01.130515 6928 catalog_manager.cc:4841] TS b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195:44687): tablet 18c1a98d2b0448cda902d2ac0cde57d3 (table test-table [id=c9cfeda1e41a40ac959fb0ef77bd2923]) successfully deleted
I20260619 22:26:01.131179 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 13a80a229cc54cd9abbc53c4aeb01a89 (DEFAULT_TABLE table=test-table [id=cefcfcd152f647959bcad3aa0db9c078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:01.131256 7470 ts_tablet_manager.cc:1905] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:01.131479 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a80a229cc54cd9abbc53c4aeb01a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:01.131644 7450 tablet_bootstrap.cc:492] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: No bootstrap required, opened a new log
I20260619 22:26:01.131734 7450 ts_tablet_manager.cc:1397] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:01.132228 7450 raft_consensus.cc:348] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.132354 7450 raft_consensus.cc:374] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:01.132386 7450 raft_consensus.cc:729] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25a28e21460405480080c74c8c9a6b9, State: Initialized, Role: FOLLOWER
I20260619 22:26:01.132521 7450 consensus_queue.cc:260] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.132809 7450 ts_tablet_manager.cc:1428] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:01.134253 7449 tablet_bootstrap.cc:492] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Bootstrap starting.
I20260619 22:26:01.135308 7449 tablet_bootstrap.cc:654] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:01.135504 7470 ts_tablet_manager.cc:1918] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:01.135620 7470 log.cc:1196] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/18c1a98d2b0448cda902d2ac0cde57d3
I20260619 22:26:01.136008 7470 ts_tablet_manager.cc:1939] T 18c1a98d2b0448cda902d2ac0cde57d3 P c25a28e21460405480080c74c8c9a6b9: Deleting consensus metadata
I20260619 22:26:01.136656 7449 tablet_bootstrap.cc:492] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: No bootstrap required, opened a new log
I20260619 22:26:01.136742 7449 ts_tablet_manager.cc:1397] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:01.137537 7449 raft_consensus.cc:348] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.137629 7449 raft_consensus.cc:374] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:01.137650 7449 raft_consensus.cc:729] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ec295f201143b3a63cb5596b25cb70, State: Initialized, Role: FOLLOWER
I20260619 22:26:01.137876 7449 consensus_queue.cc:260] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.138142 7449 ts_tablet_manager.cc:1428] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:01.138734 6928 catalog_manager.cc:4841] TS c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053): tablet 18c1a98d2b0448cda902d2ac0cde57d3 (table test-table [id=c9cfeda1e41a40ac959fb0ef77bd2923]) successfully deleted
I20260619 22:26:01.140112 7448 tablet_bootstrap.cc:492] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: No bootstrap required, opened a new log
I20260619 22:26:01.140233 7448 ts_tablet_manager.cc:1397] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260619 22:26:01.140751 7448 raft_consensus.cc:348] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.140866 7448 raft_consensus.cc:374] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:01.140889 7448 raft_consensus.cc:729] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b083c0e812df42c9bfcce2918aaaf55b, State: Initialized, Role: FOLLOWER
I20260619 22:26:01.141028 7448 consensus_queue.cc:260] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.141290 7448 ts_tablet_manager.cc:1428] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 22:26:01.234674 7458 raft_consensus.cc:482] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:01.234822 7458 raft_consensus.cc:504] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.235345 7458 leader_election.cc:288] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545), c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053)
I20260619 22:26:01.235796 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a80a229cc54cd9abbc53c4aeb01a89" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25a28e21460405480080c74c8c9a6b9" is_pre_election: true
I20260619 22:26:01.235801 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a80a229cc54cd9abbc53c4aeb01a89" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ec295f201143b3a63cb5596b25cb70" is_pre_election: true
I20260619 22:26:01.235980 7239 raft_consensus.cc:2454] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 0.
I20260619 22:26:01.235994 7099 raft_consensus.cc:2454] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 0.
I20260619 22:26:01.236322 7312 leader_election.cc:302] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b083c0e812df42c9bfcce2918aaaf55b, c25a28e21460405480080c74c8c9a6b9; no voters:
I20260619 22:26:01.236564 7458 raft_consensus.cc:2781] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:01.236634 7458 raft_consensus.cc:482] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:01.236689 7458 raft_consensus.cc:3037] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.237804 7458 raft_consensus.cc:504] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.238225 7458 leader_election.cc:288] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 election: Requested vote from peers 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545), c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053)
I20260619 22:26:01.238567 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a80a229cc54cd9abbc53c4aeb01a89" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ec295f201143b3a63cb5596b25cb70"
I20260619 22:26:01.238637 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a80a229cc54cd9abbc53c4aeb01a89" candidate_uuid: "b083c0e812df42c9bfcce2918aaaf55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25a28e21460405480080c74c8c9a6b9"
I20260619 22:26:01.238708 7099 raft_consensus.cc:3037] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.238767 7239 raft_consensus.cc:3037] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.239831 7239 raft_consensus.cc:2454] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 1.
I20260619 22:26:01.239974 7099 raft_consensus.cc:2454] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b083c0e812df42c9bfcce2918aaaf55b in term 1.
I20260619 22:26:01.240204 7312 leader_election.cc:302] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b083c0e812df42c9bfcce2918aaaf55b, c25a28e21460405480080c74c8c9a6b9; no voters:
I20260619 22:26:01.240362 7458 raft_consensus.cc:2781] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:01.240455 7458 raft_consensus.cc:686] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 1 LEADER]: Becoming Leader. State: Replica: b083c0e812df42c9bfcce2918aaaf55b, State: Running, Role: LEADER
I20260619 22:26:01.240594 7458 consensus_queue.cc:237] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.242447 6937 catalog_manager.cc:5495] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b reported cstate change: term changed from 0 to 1, leader changed from <none> to b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195). New cstate: current_term: 1 leader_uuid: "b083c0e812df42c9bfcce2918aaaf55b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } health_report { overall_health: HEALTHY } } }
Jun 19 22:26:01 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:01.293274 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.284045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.5.176.254:42519)
0619 22:26:01.284180 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.284185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.284505 (+ 320us) server_negotiation.cc:407] Connection header received
0619 22:26:01.284699 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.284704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.284769 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.284873 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:01.285724 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.286356 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.287253 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.287465 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.287539 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.289850 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:01.289859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:01.289860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:01.289880 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:01.291392 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:01.291921 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:01.291925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:01.291927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:01.292012 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:01.292314 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:01.292317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:01.292318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:01.292506 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:01.292616 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.292919 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.293080 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260619 22:26:01.293274 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.283959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42519 (local address 127.0.0.1:41674)
0619 22:26:01.284383 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.284398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.284581 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.284916 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.284919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.285498 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:01.285616 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.285623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.286470 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.286473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.287143 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.287153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.287570 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.287573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.287633 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.288180 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:01.288191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:01.289698 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:01.291526 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:01.291540 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:01.291547 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:01.291794 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:01.292110 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:01.292113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:01.292115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:01.292217 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:01.292615 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:01.292624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:01.292743 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.292876 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.293138 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":112,"thread_start_us":177,"threads_started":1}
I20260619 22:26:01.297612 6937 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:41674:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 22:26:01.298419 6937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:01.304260 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 78189e55e8584ab4a1ac1120f0564bf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b58b3913b0f748509514af1789608d22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:01.304355 7219 tablet_service.cc:1467] Processing CreateTablet for tablet 78189e55e8584ab4a1ac1120f0564bf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b58b3913b0f748509514af1789608d22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:01.304740 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78189e55e8584ab4a1ac1120f0564bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:01.304785 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78189e55e8584ab4a1ac1120f0564bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:01.307024 7359 tablet_service.cc:1467] Processing CreateTablet for tablet 78189e55e8584ab4a1ac1120f0564bf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b58b3913b0f748509514af1789608d22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:01.307411 7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78189e55e8584ab4a1ac1120f0564bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:01.307515 7450 tablet_bootstrap.cc:492] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: Bootstrap starting.
I20260619 22:26:01.307515 7449 tablet_bootstrap.cc:492] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: Bootstrap starting.
I20260619 22:26:01.308508 7450 tablet_bootstrap.cc:654] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:01.308508 7449 tablet_bootstrap.cc:654] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:01.310125 7448 tablet_bootstrap.cc:492] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: Bootstrap starting.
I20260619 22:26:01.310384 7449 tablet_bootstrap.cc:492] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: No bootstrap required, opened a new log
I20260619 22:26:01.310498 7450 tablet_bootstrap.cc:492] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: No bootstrap required, opened a new log
I20260619 22:26:01.310504 7449 ts_tablet_manager.cc:1397] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 22:26:01.310577 7450 ts_tablet_manager.cc:1397] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:01.310914 7449 raft_consensus.cc:348] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.311017 7449 raft_consensus.cc:374] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:01.311038 7449 raft_consensus.cc:729] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ec295f201143b3a63cb5596b25cb70, State: Initialized, Role: FOLLOWER
I20260619 22:26:01.311049 7448 tablet_bootstrap.cc:654] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:01.311028 7450 raft_consensus.cc:348] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.311134 7450 raft_consensus.cc:374] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:01.311167 7450 raft_consensus.cc:729] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25a28e21460405480080c74c8c9a6b9, State: Initialized, Role: FOLLOWER
I20260619 22:26:01.311192 7449 consensus_queue.cc:260] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.311290 7450 consensus_queue.cc:260] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.311517 7449 ts_tablet_manager.cc:1428] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:01.311569 7450 ts_tablet_manager.cc:1428] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:01.311604 7456 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.311751 7456 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.311798 7455 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.311936 7455 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.312739 7448 tablet_bootstrap.cc:492] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: No bootstrap required, opened a new log
I20260619 22:26:01.312836 7448 ts_tablet_manager.cc:1397] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 22:26:01.313277 7448 raft_consensus.cc:348] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.313395 7448 raft_consensus.cc:374] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:01.313436 7448 raft_consensus.cc:729] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b083c0e812df42c9bfcce2918aaaf55b, State: Initialized, Role: FOLLOWER
I20260619 22:26:01.313553 7448 consensus_queue.cc:260] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.313820 7448 ts_tablet_manager.cc:1428] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:01.314024 7458 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.314200 7458 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.586246 7456 raft_consensus.cc:482] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:01.586406 7456 raft_consensus.cc:504] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.587420 7456 leader_election.cc:288] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053), b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195:44687)
I20260619 22:26:01.592844 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.588266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:36887 (local address 127.5.176.195:44687)
0619 22:26:01.588521 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.588526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.588598 (+ 72us) server_negotiation.cc:407] Connection header received
0619 22:26:01.588661 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.588663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.588701 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.588810 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:01.589031 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.589739 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.591423 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.592036 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.592163 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.592215 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.592618 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.592699 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":62,"threads_started":1}
I20260619 22:26:01.592911 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.587746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:44053 (local address 127.5.176.193:60177)
0619 22:26:01.588024 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.588047 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.588192 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.588509 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.588513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.588536 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:01.588658 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.588666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.589702 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.589704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.591291 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.591302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.592198 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.592203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.592426 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.592488 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.592589 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.592718 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":200,"thread_start_us":77,"threads_started":1}
I20260619 22:26:01.592844 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.587847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:60177 (local address 127.5.176.194:44053)
0619 22:26:01.588148 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.588153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.588172 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:26:01.588318 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.588322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.588383 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.588485 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:01.588778 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.589596 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.591423 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.592051 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.592163 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.592215 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.592618 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.592700 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260619 22:26:01.592900 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.588140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:44687 (local address 127.5.176.193:36887)
0619 22:26:01.588498 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.588511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.588596 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.588824 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.588826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.588843 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:01.588922 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.588926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.589841 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.589844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.591291 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.591302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.592198 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.592203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.592436 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.592489 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.592591 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.592692 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":259,"thread_start_us":114,"threads_started":1}
I20260619 22:26:01.593612 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78189e55e8584ab4a1ac1120f0564bf2" candidate_uuid: "81ec295f201143b3a63cb5596b25cb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25a28e21460405480080c74c8c9a6b9" is_pre_election: true
I20260619 22:26:01.593672 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78189e55e8584ab4a1ac1120f0564bf2" candidate_uuid: "81ec295f201143b3a63cb5596b25cb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b083c0e812df42c9bfcce2918aaaf55b" is_pre_election: true
I20260619 22:26:01.593786 7239 raft_consensus.cc:2454] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81ec295f201143b3a63cb5596b25cb70 in term 0.
I20260619 22:26:01.593885 7379 raft_consensus.cc:2454] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81ec295f201143b3a63cb5596b25cb70 in term 0.
I20260619 22:26:01.594214 7032 leader_election.cc:302] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ec295f201143b3a63cb5596b25cb70, c25a28e21460405480080c74c8c9a6b9; no voters:
I20260619 22:26:01.594468 7456 raft_consensus.cc:2781] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:01.594530 7456 raft_consensus.cc:482] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:01.594569 7456 raft_consensus.cc:3037] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.595582 7456 raft_consensus.cc:504] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.595914 7456 leader_election.cc:288] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [CANDIDATE]: Term 1 election: Requested vote from peers c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053), b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195:44687)
I20260619 22:26:01.596266 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78189e55e8584ab4a1ac1120f0564bf2" candidate_uuid: "81ec295f201143b3a63cb5596b25cb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25a28e21460405480080c74c8c9a6b9"
I20260619 22:26:01.596282 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78189e55e8584ab4a1ac1120f0564bf2" candidate_uuid: "81ec295f201143b3a63cb5596b25cb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b083c0e812df42c9bfcce2918aaaf55b"
I20260619 22:26:01.596380 7239 raft_consensus.cc:3037] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.596410 7379 raft_consensus.cc:3037] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:01.597455 7239 raft_consensus.cc:2454] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81ec295f201143b3a63cb5596b25cb70 in term 1.
I20260619 22:26:01.597635 7379 raft_consensus.cc:2454] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81ec295f201143b3a63cb5596b25cb70 in term 1.
I20260619 22:26:01.597796 7032 leader_election.cc:302] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ec295f201143b3a63cb5596b25cb70, c25a28e21460405480080c74c8c9a6b9; no voters:
I20260619 22:26:01.597957 7456 raft_consensus.cc:2781] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:01.598201 7456 raft_consensus.cc:686] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [term 1 LEADER]: Becoming Leader. State: Replica: 81ec295f201143b3a63cb5596b25cb70, State: Running, Role: LEADER
I20260619 22:26:01.598467 7456 consensus_queue.cc:237] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } }
I20260619 22:26:01.599551 7490 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: TxnStatusTablet state changed. Reason: New leader 81ec295f201143b3a63cb5596b25cb70. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.599677 7490 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:01.600065 7492 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 22:26:01.600224 7492 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 22:26:01.600761 6940 catalog_manager.cc:5495] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193). New cstate: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:01.678056 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.674014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34545 (local address 127.0.0.1:47596)
0619 22:26:01.674190 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.674208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.674350 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.674703 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.674707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.674714 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:01.674830 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.674837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.675642 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.675649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.676374 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.676383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.676855 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.676859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.676955 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.677002 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.677740 (+ 738us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.677756 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.677824 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.677919 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 22:26:01.678072 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.674154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.5.176.193:34545)
0619 22:26:01.674411 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.674417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.674434 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:26:01.674511 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.674514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.674580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.674695 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:01.674937 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.675530 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.676507 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.676741 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.676860 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.677161 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.677632 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.677757 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.677841 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.677900 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260619 22:26:01.681382 7239 raft_consensus.cc:1261] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Refusing update from remote peer 81ec295f201143b3a63cb5596b25cb70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:01.681586 7379 raft_consensus.cc:1261] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Refusing update from remote peer 81ec295f201143b3a63cb5596b25cb70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:01.681870 7490 consensus_queue.cc:1035] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:01.682111 7456 consensus_queue.cc:1035] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:01.683725 7455 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: TxnStatusTablet state changed. Reason: New leader 81ec295f201143b3a63cb5596b25cb70. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.683852 7455 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.684036 7458 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: TxnStatusTablet state changed. Reason: New leader 81ec295f201143b3a63cb5596b25cb70. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.684140 7458 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.687107 7455 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.687237 7455 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P c25a28e21460405480080c74c8c9a6b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.687722 7458 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.687842 7458 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P b083c0e812df42c9bfcce2918aaaf55b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:01.687855 7456 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.687970 7456 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:01.688082 7456 tablet_replica.cc:439] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81ec295f201143b3a63cb5596b25cb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 } } peers { permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 } } peers { permanent_uuid: "b083c0e812df42c9bfcce2918aaaf55b" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44687 } } }
I20260619 22:26:01.688144 7456 tablet_replica.cc:442] T 78189e55e8584ab4a1ac1120f0564bf2 P 81ec295f201143b3a63cb5596b25cb70: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:01.704980 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.700985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:34457 (local address 127.5.176.193:34545)
0619 22:26:01.701110 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.701115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.701352 (+ 237us) server_negotiation.cc:407] Connection header received
0619 22:26:01.701489 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.701492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.701537 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.701603 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:01.701915 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.702522 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.703824 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.704048 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.704145 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.704527 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.704604 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.704674 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.704805 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.704859 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 22:26:01.705051 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.700874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34545 (local address 127.5.176.195:34457)
0619 22:26:01.701264 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.701280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.701396 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.701644 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.701647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.701657 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:01.701768 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.701775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.702650 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.702654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.703664 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.703676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.704184 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.704188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.704287 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.704349 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.704707 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.704714 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.704787 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.704883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260619 22:26:01.712291 7099 raft_consensus.cc:1261] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Refusing update from remote peer b083c0e812df42c9bfcce2918aaaf55b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:01.712296 7239 raft_consensus.cc:1261] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Refusing update from remote peer b083c0e812df42c9bfcce2918aaaf55b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:01.712677 7454 consensus_queue.cc:1035] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [LEADER]: Connected to new peer: Peer: permanent_uuid: "81ec295f201143b3a63cb5596b25cb70" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:01.712895 7458 consensus_queue.cc:1035] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25a28e21460405480080c74c8c9a6b9" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 44053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:01.735656 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.731755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:44687 (local address 127.5.176.193:44653)
0619 22:26:01.732091 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:01.732111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:01.732200 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:01.732425 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:01.732427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:01.732432 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:01.732539 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.732543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.733528 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.733531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.734341 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:01.734351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.734948 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.734951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:01.735023 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.735065 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.735389 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.735394 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 22:26:01.735440 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 22:26:01.735507 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":143,"threads_started":1}
I20260619 22:26:01.735658 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.731810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:44653 (local address 127.5.176.195:44687)
0619 22:26:01.731932 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.731936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.732197 (+ 261us) server_negotiation.cc:407] Connection header received
0619 22:26:01.732277 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.732281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.732334 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.732423 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:01.732636 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.733415 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.734508 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.734837 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.734952 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.735207 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:01.735296 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:01.735378 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.735469 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.735526 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 22:26:01.761859 6940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:01.762042 6940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:01.764458 6940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a030265df674202b801c3bd9339d79a: Sending DeleteTablet for 3 replicas of tablet 13a80a229cc54cd9abbc53c4aeb01a89
I20260619 22:26:01.765311 7219 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a80a229cc54cd9abbc53c4aeb01a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:01 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:35254
I20260619 22:26:01.765795 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a80a229cc54cd9abbc53c4aeb01a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:01 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:47582
I20260619 22:26:01.765947 7514 tablet_replica.cc:330] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: stopping tablet replica
I20260619 22:26:01.766098 7514 raft_consensus.cc:2229] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:01.766263 7514 raft_consensus.cc:2258] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:01.766553 7359 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a80a229cc54cd9abbc53c4aeb01a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:01 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:42034
I20260619 22:26:01.766779 7516 tablet_replica.cc:330] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: stopping tablet replica
I20260619 22:26:01.766950 7516 raft_consensus.cc:2229] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:01.767048 7514 ts_tablet_manager.cc:1905] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:01.767098 7516 raft_consensus.cc:2258] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:01.767794 7516 ts_tablet_manager.cc:1905] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:01.769716 7514 ts_tablet_manager.cc:1918] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:01.769843 7514 log.cc:1196] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/13a80a229cc54cd9abbc53c4aeb01a89
I20260619 22:26:01.770116 7514 ts_tablet_manager.cc:1939] T 13a80a229cc54cd9abbc53c4aeb01a89 P c25a28e21460405480080c74c8c9a6b9: Deleting consensus metadata
I20260619 22:26:01.770414 7517 tablet_replica.cc:330] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: stopping tablet replica
I20260619 22:26:01.770583 7516 ts_tablet_manager.cc:1918] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:01.770663 7516 log.cc:1196] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/13a80a229cc54cd9abbc53c4aeb01a89
I20260619 22:26:01.770612 7517 raft_consensus.cc:2229] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:01.770915 7516 ts_tablet_manager.cc:1939] T 13a80a229cc54cd9abbc53c4aeb01a89 P 81ec295f201143b3a63cb5596b25cb70: Deleting consensus metadata
I20260619 22:26:01.771003 7517 raft_consensus.cc:2258] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:01.770985 6928 catalog_manager.cc:4841] TS c25a28e21460405480080c74c8c9a6b9 (127.5.176.194:44053): tablet 13a80a229cc54cd9abbc53c4aeb01a89 (table test-table [id=cefcfcd152f647959bcad3aa0db9c078]) successfully deleted
I20260619 22:26:01.771584 6927 catalog_manager.cc:4841] TS 81ec295f201143b3a63cb5596b25cb70 (127.5.176.193:34545): tablet 13a80a229cc54cd9abbc53c4aeb01a89 (table test-table [id=cefcfcd152f647959bcad3aa0db9c078]) successfully deleted
I20260619 22:26:01.771770 7517 ts_tablet_manager.cc:1905] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:01.774967 7517 ts_tablet_manager.cc:1918] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:01.775056 7517 log.cc:1196] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/13a80a229cc54cd9abbc53c4aeb01a89
I20260619 22:26:01.775410 7517 ts_tablet_manager.cc:1939] T 13a80a229cc54cd9abbc53c4aeb01a89 P b083c0e812df42c9bfcce2918aaaf55b: Deleting consensus metadata
I20260619 22:26:01.776152 6928 catalog_manager.cc:4841] TS b083c0e812df42c9bfcce2918aaaf55b (127.5.176.195:44687): tablet 13a80a229cc54cd9abbc53c4aeb01a89 (table test-table [id=cefcfcd152f647959bcad3aa0db9c078]) successfully deleted
Jun 19 22:26:01 dist-test-slave-bt27 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:26:01.785171 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.767138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.5.176.193:34545)
0619 22:26:01.767253 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.767258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.774451 (+ 7193us) server_negotiation.cc:407] Connection header received
0619 22:26:01.774684 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.774687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.774749 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.774822 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:01.776129 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.776715 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.777634 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.777913 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.778013 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.780997 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:01.781008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:01.781022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:01.781047 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:01.783046 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:01.783547 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:01.783554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:01.783556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:01.783647 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:01.784051 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:01.784054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:01.784056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:01.784364 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:01.784583 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.784877 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.785008 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 22:26:01.785758 7139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47604
I20260619 22:26:01.795027 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:01.787506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.5.176.254:42519)
0619 22:26:01.787632 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:01.787636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:01.787666 (+ 30us) server_negotiation.cc:407] Connection header received
0619 22:26:01.787831 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:01.787834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:01.787889 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:01.787979 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:01.788600 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.789126 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.789924 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:01.790247 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:01.790351 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:01.791767 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:01.791773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:01.791777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:01.791806 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:01.793300 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:01.793726 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:01.793731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:01.793732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:01.793822 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:01.794101 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:01.794106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:01.794108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:01.794359 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:01.794502 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:01.794733 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 22:26:01.794861 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 22:26:01.795437 6940 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41688
I20260619 22:26:01.796540 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7014
I20260619 22:26:01.805765 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7154
I20260619 22:26:01.812213 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7294
I20260619 22:26:01.818771 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 6910
2026-06-19T22:26:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3167 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:01 dist-test-slave-bt27 krb5kdc[7523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:01 dist-test-slave-bt27 krb5kdc[7523](info): set up 2 sockets
Jun 19 22:26:01 dist-test-slave-bt27 krb5kdc[7523](info): commencing operation
krb5kdc: starting...
W20260619 22:26:02.874007 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:02 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:03.001163 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43361
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:43361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:03.112330 7536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:03.112634 7536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:03.112676 7536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:03.116806 7536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:03.116882 7536 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 22:26:03.116900 7536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:03.116918 7536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:03.116935 7536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:03.116950 7536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:03.122474 7536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:43361
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:03.123829 7536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:03.124964 7536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:03.134475 7541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:03.134617 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.134737 7536 server_base.cc:1034] running on GCE node
W20260619 22:26:03.134475 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.135608 7536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:03.137115 7536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:03.138343 7536 hybrid_clock.cc:648] HybridClock initialized: now 1781907963138320 us; error 40 us; skew 500 ppm
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:03.145457 7536 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:03.146733 7536 webserver.cc:460] Webserver started at http://127.5.176.254:39059/ using document root <none> and password file <none>
I20260619 22:26:03.147374 7536 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:03.147475 7536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:03.147711 7536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:03.149991 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "e3ab4accd0464fd3ab61e65f2da9e914"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "140322c56209274a3f2c99827999d2a3"
server_key_iv: "0b5b9815f05dc29ab75032ed80640aac"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.150692 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "e3ab4accd0464fd3ab61e65f2da9e914"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "140322c56209274a3f2c99827999d2a3"
server_key_iv: "0b5b9815f05dc29ab75032ed80640aac"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.154680 7536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:03.157539 7551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:03.158960 7536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:03.159102 7536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "e3ab4accd0464fd3ab61e65f2da9e914"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "140322c56209274a3f2c99827999d2a3"
server_key_iv: "0b5b9815f05dc29ab75032ed80640aac"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.159228 7536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:03.177193 7536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:03.181893 7536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:03.182204 7536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:03.191215 7536 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:43361
I20260619 22:26:03.191231 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:43361 every 8 connection(s)
I20260619 22:26:03.192239 7536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:03.195384 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:03.198258 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7536
I20260619 22:26:03.198375 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:03.198643 5827 external_mini_cluster.cc:1428] Setting key 3e2908ef48230d601506b3a853b3f889
I20260619 22:26:03.201997 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Bootstrap starting.
I20260619 22:26:03.204573 7614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:03.205523 7614 log.cc:824] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:03.208112 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: No bootstrap required, opened a new log
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:03.211628 7614 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43361 } }
I20260619 22:26:03.211892 7614 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:03.211977 7614 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ab4accd0464fd3ab61e65f2da9e914, State: Initialized, Role: FOLLOWER
I20260619 22:26:03.212523 7614 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43361 } }
I20260619 22:26:03.212685 7614 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:03.212771 7614 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:03.212881 7614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:03.213861 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.200228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.5.176.254:43361)
0619 22:26:03.200753 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:03.200769 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:03.200807 (+ 38us) server_negotiation.cc:407] Connection header received
0619 22:26:03.201476 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:03.201494 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:03.201856 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:03.202259 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:03.203070 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.204267 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.205261 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.205930 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.206124 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.209000 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:03.209005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:03.209022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:03.209050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:03.210864 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.211369 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.211375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.211380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.211492 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.211790 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.211794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.211796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.212306 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:03.212528 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:03.212765 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 22:26:03.213091 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":151,"threads_started":1}
I20260619 22:26:03.214051 7614 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43361 } }
I20260619 22:26:03.214447 7614 leader_election.cc:302] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ab4accd0464fd3ab61e65f2da9e914; no voters:
I20260619 22:26:03.214857 7614 leader_election.cc:288] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:03.214929 7619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:03.215116 7619 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [term 1 LEADER]: Becoming Leader. State: Replica: e3ab4accd0464fd3ab61e65f2da9e914, State: Running, Role: LEADER
I20260619 22:26:03.215505 7619 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43361 } }
I20260619 22:26:03.216209 7614 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:03.216717 7621 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43361 } } }
I20260619 22:26:03.216873 7621 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:03.216975 7620 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ab4accd0464fd3ab61e65f2da9e914. Latest consensus state: current_term: 1 leader_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ab4accd0464fd3ab61e65f2da9e914" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43361 } } }
I20260619 22:26:03.217324 7620 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:03.217412 7625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:03.221050 7625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:03.227411 7625 catalog_manager.cc:1347] Generated new cluster ID: ef989a3d711f4c44b1f04c548ebe535a
I20260619 22:26:03.227509 7625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:03.268433 7625 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:03.269435 7625 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:03.282226 7625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Generated new TSK 0
I20260619 22:26:03.283001 7625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:03.323052 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43361
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:03.455302 7640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:03.455593 7640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:03.455642 7640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:03.460623 7640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:03.460692 7640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:03.460733 7640 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 22:26:03.460748 7640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:03.460811 7640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:03.467100 7640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:43361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:03.468274 7640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:03.469331 7640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:03.479029 7646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.479180 7640 server_base.cc:1034] running on GCE node
W20260619 22:26:03.479018 7645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:03.479025 7648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.479883 7640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:03.480561 7640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:03.481777 7640 hybrid_clock.cc:648] HybridClock initialized: now 1781907963481741 us; error 35 us; skew 500 ppm
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:03.484810 7640 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:03.485697 7640 webserver.cc:460] Webserver started at http://127.5.176.193:36771/ using document root <none> and password file <none>
I20260619 22:26:03.486326 7640 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:03.486398 7640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:03.486579 7640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:03.488780 7640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "c26cb41fa87cc9eed99d18a7ef8842d9"
server_key_iv: "c7a889a7314b71802eec5449a593c81a"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.489341 7640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "c26cb41fa87cc9eed99d18a7ef8842d9"
server_key_iv: "c7a889a7314b71802eec5449a593c81a"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.493305 7640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:26:03.495930 7655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:03.497347 7640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:03.497468 7640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "c26cb41fa87cc9eed99d18a7ef8842d9"
server_key_iv: "c7a889a7314b71802eec5449a593c81a"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.497627 7640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:03.531456 7640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:03.534605 7640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:03.534833 7640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:03.536371 7640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:03.536432 7640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:03.536513 7640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:03.536556 7640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:03.548041 7640 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:39453
I20260619 22:26:03.548110 7770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:39453 every 8 connection(s)
I20260619 22:26:03.549214 7640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:03.550990 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7640
I20260619 22:26:03.551122 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:03.551477 5827 external_mini_cluster.cc:1428] Setting key e8469e358256e3c4f3b7328dc5a268f3
I20260619 22:26:03.551657 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.537429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47547 (local address 127.5.176.254:43361)
0619 22:26:03.537562 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:03.537565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:03.538707 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 22:26:03.539825 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:03.539829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:03.539893 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:03.539959 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:03.541882 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.542668 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.544163 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.544656 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.544785 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.547775 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:03.547779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:03.547783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:03.547816 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:03.549367 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.549954 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.549958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.549960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.550048 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.550412 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.550416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.550418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.550612 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:03.550723 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:03.551329 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 22:26:03.551477 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:26:03.552466 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.538007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.5.176.193:47547)
0619 22:26:03.538557 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:03.538601 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:03.539573 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:03.540417 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:03.540429 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:03.540982 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:03.541702 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.541726 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.542801 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.542804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.544011 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.544022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.544814 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.544817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.544958 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.545606 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:03.545627 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:03.547595 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:03.549485 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.549491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.549506 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.549834 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.550127 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.550133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.550158 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.550308 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.550749 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:03.550755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:03.550994 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 22:26:03.551218 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 22:26:03.551491 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":156,"threads_started":1}
I20260619 22:26:03.553960 7771 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43361
I20260619 22:26:03.554133 7771 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:03.554767 7771 heartbeater.cc:511] Master 127.5.176.254:43361 requested a full tablet report, sending...
I20260619 22:26:03.556529 7565 ts_manager.cc:194] Registered new tserver with Master: 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453)
I20260619 22:26:03.558243 7565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:47547
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:03.566929 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.558789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.5.176.193:36491)
0619 22:26:03.559173 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:03.559186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:03.559313 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:03.559601 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:03.559604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:03.559873 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:03.559996 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.560004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.561107 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.561109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.561724 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.561732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.562380 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.562383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.562469 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.563076 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:03.563092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:03.563470 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:03.565299 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.565305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.565310 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.565584 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.565909 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.565914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.565919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.566016 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.566515 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:03.566517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:03.566591 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 22:26:03.566675 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:26:03.566780 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":116,"threads_started":1}
I20260619 22:26:03.567236 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.558823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:36491 (local address 127.5.176.254:43361)
0619 22:26:03.559072 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:03.559077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:03.559276 (+ 199us) server_negotiation.cc:407] Connection header received
0619 22:26:03.559403 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:03.559408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:03.559467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:03.559560 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:03.560135 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.560971 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.561892 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.562268 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.562354 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.563607 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:03.563610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:03.563612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:03.563632 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:03.565184 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.565684 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.565686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.565687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.565793 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.566122 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.566127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.566130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.566404 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:03.566523 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:03.566758 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 22:26:03.566894 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:03.581806 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43361
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:03.705229 7780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:03.705509 7780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:03.705557 7780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:03.709865 7780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:03.709926 7780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:03.709968 7780 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 22:26:03.710012 7780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:03.710075 7780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:03.715595 7780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:43361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:03.716841 7780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:03.718045 7780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:03.727910 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:03.728012 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:03.727919 7788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.728148 7780 server_base.cc:1034] running on GCE node
I20260619 22:26:03.728623 7780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:03.729245 7780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:03.730412 7780 hybrid_clock.cc:648] HybridClock initialized: now 1781907963730397 us; error 25 us; skew 500 ppm
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:03.733393 7780 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:03.734409 7780 webserver.cc:460] Webserver started at http://127.5.176.194:35465/ using document root <none> and password file <none>
I20260619 22:26:03.734997 7780 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:03.735090 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:03.735325 7780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:03.737432 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "e3c618f3e40b46d4a3988e5adf58f576"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "55e4303b648529e84e63d8b48da2ee47"
server_key_iv: "e734eb18a459123e0b1c8fa380cacb6f"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.738032 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "e3c618f3e40b46d4a3988e5adf58f576"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "55e4303b648529e84e63d8b48da2ee47"
server_key_iv: "e734eb18a459123e0b1c8fa380cacb6f"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.742049 7780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 22:26:03.744725 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:03.746095 7780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:03.746289 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "e3c618f3e40b46d4a3988e5adf58f576"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "55e4303b648529e84e63d8b48da2ee47"
server_key_iv: "e734eb18a459123e0b1c8fa380cacb6f"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.746429 7780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:03.782051 7780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:03.786026 7780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:03.786301 7780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:03.787803 7780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:03.787889 7780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:03.787964 7780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:03.788018 7780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:03.799486 7780 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:35485
I20260619 22:26:03.800639 7780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:03.801972 7910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:35485 every 8 connection(s)
I20260619 22:26:03.802654 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.788883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:39469 (local address 127.5.176.254:43361)
0619 22:26:03.789017 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:03.789021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:03.790204 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 22:26:03.791350 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:03.791354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:03.791412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:03.791516 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:03.793068 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.793966 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.795437 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.795674 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.795806 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.798573 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:03.798577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:03.798580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:03.798610 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:03.800295 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.800828 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.800831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.800833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.800924 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.801403 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.801406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.801408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.801647 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:03.801740 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:03.802431 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 22:26:03.802534 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:26:03.803534 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.789479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.5.176.194:39469)
0619 22:26:03.789998 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:03.790041 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:03.791126 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:03.791698 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:03.791709 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:03.792142 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:03.792922 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.792943 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.794086 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.794090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.795309 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.795318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.795853 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.795857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.796061 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.796708 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:03.796724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:03.798394 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:03.800411 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.800416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.800427 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.800670 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.801081 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.801087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.801092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.801283 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.801826 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:03.801833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:03.802185 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 22:26:03.802349 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 22:26:03.802670 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20260619 22:26:03.804894 7911 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43361
I20260619 22:26:03.805039 7911 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:03.805634 7911 heartbeater.cc:511] Master 127.5.176.254:43361 requested a full tablet report, sending...
I20260619 22:26:03.806906 7568 ts_manager.cc:194] Registered new tserver with Master: e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485)
I20260619 22:26:03.807652 7568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:39469
I20260619 22:26:03.809785 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7780
I20260619 22:26:03.809918 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:03.810235 5827 external_mini_cluster.cc:1428] Setting key 7fce1a114eaf03c26449f29ea788c46d
I20260619 22:26:03.815351 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.807120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.5.176.194:34673)
0619 22:26:03.807435 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:03.807447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:03.807580 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:03.807838 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:03.807841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:03.808065 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:03.808158 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.808164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.808913 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.808916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.809527 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:03.809538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.810058 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.810062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:03.810190 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.811026 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:03.811042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:03.811535 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:03.813365 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.813372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.813378 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.813667 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.814044 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:03.814046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:03.814048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:03.814130 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:03.814765 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:03.814768 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:03.814872 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 22:26:03.814962 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 22:26:03.815090 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
I20260619 22:26:03.815332 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:03.807164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:34673 (local address 127.5.176.254:43361)
0619 22:26:03.807317 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:03.807322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:03.807536 (+ 214us) server_negotiation.cc:407] Connection header received
0619 22:26:03.807674 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:03.807677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:03.807746 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:03.807824 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:03.808274 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.808780 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.809667 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:03.809934 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:03.810055 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:03.811679 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:03.811685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:03.811690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:03.811717 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:03.813233 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.813807 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.813813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.813817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.813934 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:03.814296 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:03.814302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:03.814305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:03.814640 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:03.814755 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:03.815052 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 22:26:03.815190 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:03.839097 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43361
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:03.950578 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:03.950961 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:03.951059 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:03.955380 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:03.955463 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:03.955533 7920 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 22:26:03.955578 7920 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:03.955644 7920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:03.961100 7920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:43361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:03.962266 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:03.963418 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:03.973351 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:03.973354 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.973456 7920 server_base.cc:1034] running on GCE node
W20260619 22:26:03.973387 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:03.974030 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:03.974723 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:03.975934 7920 hybrid_clock.cc:648] HybridClock initialized: now 1781907963975901 us; error 41 us; skew 500 ppm
Jun 19 22:26:03 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:03.978698 7920 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:03.979553 7920 webserver.cc:460] Webserver started at http://127.5.176.195:42981/ using document root <none> and password file <none>
I20260619 22:26:03.980160 7920 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:03.980258 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:03.980522 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:03.982662 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "92b0194ba994c8dd22c418277ccaa40a"
server_key_iv: "c005a759f3b03aa28c8488f1403b69b5"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.983207 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "92b0194ba994c8dd22c418277ccaa40a"
server_key_iv: "c005a759f3b03aa28c8488f1403b69b5"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.986945 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:03.989365 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:03.990700 7920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:03.990839 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
format_stamp: "Formatted at 2026-06-19 22:26:03 on dist-test-slave-bt27"
server_key: "92b0194ba994c8dd22c418277ccaa40a"
server_key_iv: "c005a759f3b03aa28c8488f1403b69b5"
server_key_version: "encryptionkey@0"
I20260619 22:26:03.990944 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:04.044154 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:04.047633 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:04.047857 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:04.049319 7920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:04.049365 7920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:04.049434 7920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:04.049467 7920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:04 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:04.060211 7920 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:43799
I20260619 22:26:04.060307 8050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:43799 every 8 connection(s)
I20260619 22:26:04.061123 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:04.064267 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.050451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:45759 (local address 127.5.176.254:43361)
0619 22:26:04.050581 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.050586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.051588 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 22:26:04.052935 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.052938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.052999 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.053096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:04.054814 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.055434 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.057281 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.057498 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.057583 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.060526 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:04.060530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:04.060538 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:04.060563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:04.062340 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.062877 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.062880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.062881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.062955 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.063252 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.063254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.063255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.063426 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:04.063502 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.064022 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.064137 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:26:04.065069 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.050894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.5.176.195:45759)
0619 22:26:04.051407 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.051463 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.052704 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.053315 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.053323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.053818 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:04.054629 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.054651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.055576 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.055581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.057155 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.057165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.057672 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.057676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.057885 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.058664 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:04.058685 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:04.060361 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:04.062448 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:04.062452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:04.062464 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:04.062784 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:04.063032 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:04.063034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:04.063036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:04.063169 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:04.063510 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:04.063515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:04.063785 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.063939 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.064182 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":149,"threads_started":1}
I20260619 22:26:04.066378 8051 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43361
I20260619 22:26:04.066392 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 7920
I20260619 22:26:04.066531 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:04.066545 8051 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:04.066859 5827 external_mini_cluster.cc:1428] Setting key b89a336183bee2f708ee320d56e08e20
I20260619 22:26:04.067152 8051 heartbeater.cc:511] Master 127.5.176.254:43361 requested a full tablet report, sending...
I20260619 22:26:04.068257 7565 ts_manager.cc:194] Registered new tserver with Master: 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.069099 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:26:04.069116 7565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:45759
I20260619 22:26:04.077128 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.069027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:49555 (local address 127.5.176.254:43361)
0619 22:26:04.069411 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.069415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.069426 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:26:04.069465 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.069467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.069497 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.069588 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:04.070070 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.070847 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.071818 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.072058 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.072174 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.073472 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:04.073475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:04.073477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:04.073501 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:04.075124 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.075676 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.075679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.075681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.075764 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.076186 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.076192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.076196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.076414 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:04.076495 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.076897 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.077004 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260619 22:26:04.077231 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.068741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.5.176.195:49555)
0619 22:26:04.069198 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.069212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.069296 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.069604 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.069607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.069827 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:04.069939 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.069946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.071005 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.071008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.071688 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.071696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.072177 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.072180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.072285 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.072801 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:04.072821 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:04.073293 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:04.075255 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:04.075262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:04.075267 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:04.075543 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:04.075933 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:04.075936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:04.075938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:04.076035 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:04.076517 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:04.076523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:04.076647 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.076795 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.076991 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":84,"threads_started":1}
Jun 19 22:26:04 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:26:04 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:04.097518 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.088937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.5.176.254:43361)
0619 22:26:04.089056 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.089059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.089071 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:04.089110 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.089113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.089151 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.089261 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:04.090013 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.090645 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.091538 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.091765 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.091882 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.094530 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:04.094534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:04.094536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:04.094564 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:04.095956 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.096368 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.096371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.096372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.096455 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.096695 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.096697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.096699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.096877 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:04.096980 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.097174 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.097288 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 22:26:04.101100 7565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:04.104386 7565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:04.120000 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.114499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.5.176.194:35485)
0619 22:26:04.114836 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.114847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.114990 (+ 143us) server_negotiation.cc:407] Connection header received
0619 22:26:04.115176 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.115181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.115376 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.115525 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:04.115861 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.116728 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.118659 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.119231 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.119374 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.119541 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.119719 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.119821 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
I20260619 22:26:04.119997 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.113539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:35485 (local address 127.0.0.1:46652)
0619 22:26:04.114902 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.114917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.115021 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.115505 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.115507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.115527 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:04.115714 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.115726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.116852 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.116855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.118532 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.118538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.119372 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.119376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.119493 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.119531 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.119677 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.119831 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":91,"thread_start_us":129,"threads_started":1}
I20260619 22:26:04.120049 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.113363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39453 (local address 127.0.0.1:52282)
0619 22:26:04.114240 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.114295 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.114425 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.115052 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.115070 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.115097 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:04.115252 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.115260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.117129 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.117133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.118513 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.118524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.119603 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.119606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.119698 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.119721 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.119774 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.119865 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":46,"thread_start_us":78,"threads_started":1}
I20260619 22:26:04.120342 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.114256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.5.176.195:43799)
0619 22:26:04.115317 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.115329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.115433 (+ 104us) server_negotiation.cc:407] Connection header received
0619 22:26:04.115596 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.115601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.115769 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.115947 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:04.116185 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.117282 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.118691 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.119461 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.119578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.119714 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.120079 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.120181 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":113,"threads_started":1}
I20260619 22:26:04.120446 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.113793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:43799 (local address 127.0.0.1:57176)
0619 22:26:04.115327 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.115341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.115436 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.115933 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.115936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.115954 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:04.116056 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.116064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.117412 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.117415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.118537 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.118547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.119771 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.119774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.119868 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.119887 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.119954 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.120012 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"mutex_wait_us":43,"thread_start_us":117,"threads_started":1}
I20260619 22:26:04.121146 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.114020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.5.176.193:39453)
0619 22:26:04.114540 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.114550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.114572 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:26:04.114673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.114678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.114901 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.115054 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:04.115419 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.116823 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.118644 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.119461 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.120229 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.120863 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.120914 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.121011 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":71,"threads_started":1}
I20260619 22:26:04.123622 7845 tablet_service.cc:1467] Processing CreateTablet for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c (DEFAULT_TABLE table=test-table [id=6f867cd90a234b34bd6acba3a093f29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:04.123654 7701 tablet_service.cc:1467] Processing CreateTablet for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c (DEFAULT_TABLE table=test-table [id=6f867cd90a234b34bd6acba3a093f29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:04.123643 7985 tablet_service.cc:1467] Processing CreateTablet for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c (DEFAULT_TABLE table=test-table [id=6f867cd90a234b34bd6acba3a093f29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:04.124735 7845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.124737 7985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.124737 7701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.132647 8074 tablet_bootstrap.cc:492] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Bootstrap starting.
I20260619 22:26:04.133116 8075 tablet_bootstrap.cc:492] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Bootstrap starting.
I20260619 22:26:04.134022 8076 tablet_bootstrap.cc:492] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Bootstrap starting.
I20260619 22:26:04.135663 8075 tablet_bootstrap.cc:654] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.136207 8076 tablet_bootstrap.cc:654] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.136613 8075 log.cc:824] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:04.137022 8076 log.cc:824] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:04.139410 8076 tablet_bootstrap.cc:492] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: No bootstrap required, opened a new log
I20260619 22:26:04.139429 8074 tablet_bootstrap.cc:654] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.139640 8076 ts_tablet_manager.cc:1397] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:26:04.140272 8074 log.cc:824] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:04.142547 8074 tablet_bootstrap.cc:492] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: No bootstrap required, opened a new log
I20260619 22:26:04.142772 8074 ts_tablet_manager.cc:1397] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 22:26:04.143709 8075 tablet_bootstrap.cc:492] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: No bootstrap required, opened a new log
I20260619 22:26:04.143946 8075 ts_tablet_manager.cc:1397] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260619 22:26:04.143734 8076 raft_consensus.cc:348] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.144034 8076 raft_consensus.cc:374] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.144127 8076 raft_consensus.cc:729] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.144661 8076 consensus_queue.cc:260] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.145874 8076 ts_tablet_manager.cc:1428] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:26:04.146309 7911 heartbeater.cc:503] Master 127.5.176.254:43361 was elected leader, sending a full tablet report...
I20260619 22:26:04.147100 8074 raft_consensus.cc:348] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.147363 8074 raft_consensus.cc:374] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.147459 8074 raft_consensus.cc:729] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f1a05285742e4bb4d4ff762b958c3, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.147861 8075 raft_consensus.cc:348] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.148124 8075 raft_consensus.cc:374] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.148064 8074 consensus_queue.cc:260] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.148219 8075 raft_consensus.cc:729] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1991ec6e604daf9d9a77c3df9a6dac, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.148782 7771 heartbeater.cc:503] Master 127.5.176.254:43361 was elected leader, sending a full tablet report...
I20260619 22:26:04.148722 8075 consensus_queue.cc:260] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.149160 8074 ts_tablet_manager.cc:1428] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:26:04.149715 8075 ts_tablet_manager.cc:1428] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:04.149739 8051 heartbeater.cc:503] Master 127.5.176.254:43361 was elected leader, sending a full tablet report...
W20260619 22:26:04.300877 7772 tablet.cc:2369] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:04.302258 7912 tablet.cc:2369] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:04.312325 8052 tablet.cc:2369] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:04.505009 8080 raft_consensus.cc:482] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:04.505273 8080 raft_consensus.cc:504] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.506577 8080 leader_election.cc:288] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.510630 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.506998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:51759 (local address 127.5.176.195:43799)
0619 22:26:04.507113 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.507118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.507404 (+ 286us) server_negotiation.cc:407] Connection header received
0619 22:26:04.507536 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.507539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.507591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.507672 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:04.507936 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.508425 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.509550 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.509984 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.510084 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.510212 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.510394 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.510487 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 22:26:04.510675 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.506897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:43799 (local address 127.5.176.194:51759)
0619 22:26:04.507236 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.507249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.507406 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.507690 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.507692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.507706 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:04.507818 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.507824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.508566 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.508569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.509405 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.509414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.510103 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.510109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.510258 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.510281 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.510363 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.510466 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":37,"thread_start_us":80,"threads_started":1}
I20260619 22:26:04.510766 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.506891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:46429 (local address 127.5.176.193:39453)
0619 22:26:04.507064 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.507069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.507206 (+ 137us) server_negotiation.cc:407] Connection header received
0619 22:26:04.507381 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.507385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.507434 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.507520 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:04.507828 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.508386 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.509552 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.510062 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.510231 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.510282 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.510501 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.510587 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 22:26:04.510687 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.506764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39453 (local address 127.5.176.194:46429)
0619 22:26:04.507075 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.507091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.507216 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.507536 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.507539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.507558 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:04.507701 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.507708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.508500 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.508507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.509411 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.509420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.510233 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.510238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.510353 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.510384 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.510462 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.510529 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":22,"thread_start_us":76,"threads_started":1}
I20260619 22:26:04.511489 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee21cb5cba451eaf8ec6d4f4a42f3c" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" is_pre_election: true
I20260619 22:26:04.511488 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee21cb5cba451eaf8ec6d4f4a42f3c" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" is_pre_election: true
I20260619 22:26:04.511808 7998 raft_consensus.cc:2454] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:04.511819 7725 raft_consensus.cc:2454] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:04.512333 7798 leader_election.cc:302] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, 9f3f1a05285742e4bb4d4ff762b958c3, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:04.512622 8080 raft_consensus.cc:2781] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:04.512696 8080 raft_consensus.cc:482] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:04.512727 8080 raft_consensus.cc:3037] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.513971 8080 raft_consensus.cc:504] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.514488 8080 leader_election.cc:288] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Requested vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.515077 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee21cb5cba451eaf8ec6d4f4a42f3c" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
I20260619 22:26:04.515165 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee21cb5cba451eaf8ec6d4f4a42f3c" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
I20260619 22:26:04.515240 7998 raft_consensus.cc:3037] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.515322 7725 raft_consensus.cc:3037] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.516635 7998 raft_consensus.cc:2454] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:04.516722 7725 raft_consensus.cc:2454] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:04.517161 7798 leader_election.cc:302] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:04.517359 8080 raft_consensus.cc:2781] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:04.517592 8080 raft_consensus.cc:686] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Becoming Leader. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Running, Role: LEADER
I20260619 22:26:04.517979 8080 consensus_queue.cc:237] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.521322 7565 catalog_manager.cc:5495] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } health_report { overall_health: UNKNOWN } } }
Jun 19 22:26:04 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:04.605003 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.596151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43361 (local address 127.0.0.1:42586)
0619 22:26:04.596500 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.596514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.596621 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.596856 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.596858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.597262 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:04.597357 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.597376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.598211 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.598214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.598777 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.598785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.599236 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.599240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.599318 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.599957 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:04.599970 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:04.601532 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:04.603351 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:04.603370 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:04.603377 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:04.603619 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:04.603907 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:04.603913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:04.603915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:04.603998 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:04.604391 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:04.604401 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:04.604531 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.604664 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.604801 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":162,"threads_started":1}
I20260619 22:26:04.605063 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.596226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.5.176.254:43361)
0619 22:26:04.596396 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.596401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.596601 (+ 200us) server_negotiation.cc:407] Connection header received
0619 22:26:04.596705 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.596707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.596749 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.596818 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:04.597513 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.598065 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.598907 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.599120 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.599231 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.601677 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:04.601681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:04.601684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:04.601711 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:04.603196 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.603726 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.603731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.603732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.603810 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:04.604093 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:04.604096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:04.604098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:04.604279 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:04.604385 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.604753 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.604868 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 22:26:04.609189 7565 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:42586:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 22:26:04.610016 7565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:04.615351 7701 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5eff774a0433cb726d5dbef0cdb83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b03ee18e8e34b3ebfbfe08bd1836dd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:04.615515 7985 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5eff774a0433cb726d5dbef0cdb83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b03ee18e8e34b3ebfbfe08bd1836dd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:04.615525 7845 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5eff774a0433cb726d5dbef0cdb83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b03ee18e8e34b3ebfbfe08bd1836dd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:04.616184 7985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5eff774a0433cb726d5dbef0cdb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.616184 7845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5eff774a0433cb726d5dbef0cdb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.616194 7701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5eff774a0433cb726d5dbef0cdb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.619220 8074 tablet_bootstrap.cc:492] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: Bootstrap starting.
I20260619 22:26:04.619623 8075 tablet_bootstrap.cc:492] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: Bootstrap starting.
I20260619 22:26:04.620225 8074 tablet_bootstrap.cc:654] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.620606 8075 tablet_bootstrap.cc:654] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.621238 8076 tablet_bootstrap.cc:492] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: Bootstrap starting.
I20260619 22:26:04.621742 8074 tablet_bootstrap.cc:492] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: No bootstrap required, opened a new log
I20260619 22:26:04.621763 8075 tablet_bootstrap.cc:492] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: No bootstrap required, opened a new log
I20260619 22:26:04.621840 8074 ts_tablet_manager.cc:1397] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:04.621867 8075 ts_tablet_manager.cc:1397] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:04.622346 8075 raft_consensus.cc:348] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.622483 8075 raft_consensus.cc:374] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.622412 8074 raft_consensus.cc:348] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.622524 8075 raft_consensus.cc:729] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1991ec6e604daf9d9a77c3df9a6dac, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.622534 8074 raft_consensus.cc:374] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.622567 8074 raft_consensus.cc:729] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f1a05285742e4bb4d4ff762b958c3, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.622694 8076 tablet_bootstrap.cc:654] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.622676 8075 consensus_queue.cc:260] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.622707 8074 consensus_queue.cc:260] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.623004 8074 ts_tablet_manager.cc:1428] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:04.622999 8081 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.623150 8081 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.623541 8075 ts_tablet_manager.cc:1428] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:04.623750 8082 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.623893 8082 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.624730 8076 tablet_bootstrap.cc:492] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: No bootstrap required, opened a new log
I20260619 22:26:04.624855 8076 ts_tablet_manager.cc:1397] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 22:26:04.625366 8076 raft_consensus.cc:348] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.625491 8076 raft_consensus.cc:374] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.625531 8076 raft_consensus.cc:729] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.625653 8076 consensus_queue.cc:260] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.625910 8076 ts_tablet_manager.cc:1428] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:04.625875 8080 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.625993 8080 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.654248 8080 raft_consensus.cc:482] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:04.654402 8080 raft_consensus.cc:504] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.654922 8080 leader_election.cc:288] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.655351 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5eff774a0433cb726d5dbef0cdb83" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" is_pre_election: true
I20260619 22:26:04.655354 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5eff774a0433cb726d5dbef0cdb83" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" is_pre_election: true
I20260619 22:26:04.655517 7998 raft_consensus.cc:2454] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:04.655519 7725 raft_consensus.cc:2454] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:04.655818 7798 leader_election.cc:302] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:04.655989 8080 raft_consensus.cc:2781] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:04.656051 8080 raft_consensus.cc:482] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:04.656097 8080 raft_consensus.cc:3037] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.657099 8080 raft_consensus.cc:504] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.657435 8080 leader_election.cc:288] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Requested vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.657829 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5eff774a0433cb726d5dbef0cdb83" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
I20260619 22:26:04.657851 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5eff774a0433cb726d5dbef0cdb83" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
I20260619 22:26:04.657943 7725 raft_consensus.cc:3037] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.658023 7998 raft_consensus.cc:3037] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.659050 7998 raft_consensus.cc:2454] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:04.659415 7798 leader_election.cc:302] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:04.659471 7725 raft_consensus.cc:2454] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:04.659601 8080 raft_consensus.cc:2781] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:04.659685 8080 raft_consensus.cc:686] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Becoming Leader. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Running, Role: LEADER
I20260619 22:26:04.659828 8080 consensus_queue.cc:237] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.660511 8085 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: TxnStatusTablet state changed. Reason: New leader e3c618f3e40b46d4a3988e5adf58f576. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.660621 8085 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:04.660934 8097 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 22:26:04.661180 8097 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 22:26:04.661643 7567 catalog_manager.cc:5495] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:04.675689 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.671762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.5.176.194:35485)
0619 22:26:04.672075 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.672086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.672100 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:04.672170 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.672172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.672222 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.672341 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:04.672624 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.673183 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.674057 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.674376 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.674490 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.674797 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:04.675229 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:04.675354 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.675451 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.675509 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260619 22:26:04.675666 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.671655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:35485 (local address 127.0.0.1:46666)
0619 22:26:04.671818 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.671834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.671974 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.672366 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.672370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.672378 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:04.672496 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.672501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.673302 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.673309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.673929 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.673938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.674495 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.674498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.674591 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.674639 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:04.675340 (+ 701us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:04.675356 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.675438 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.675535 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 22:26:04.681084 7725 raft_consensus.cc:1261] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:04.681094 7998 raft_consensus.cc:1261] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:04.681826 8085 consensus_queue.cc:1035] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:04.682020 8080 consensus_queue.cc:1035] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:04.686450 8082 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: TxnStatusTablet state changed. Reason: New leader e3c618f3e40b46d4a3988e5adf58f576. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.686594 8082 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.687986 8081 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: TxnStatusTablet state changed. Reason: New leader e3c618f3e40b46d4a3988e5adf58f576. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.688143 8081 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.689929 8080 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.690057 8080 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:04.690619 8101 mvcc.cc:205] Tried to move back new op lower bound from 7298695023326388224 to 7298695023247941632. Current Snapshot: MvccSnapshot[applied={T|T < 7298695023326388224}]
I20260619 22:26:04.691501 8085 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.691618 8085 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P e3c618f3e40b46d4a3988e5adf58f576: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:04.691805 8082 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.691910 8082 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P 3d1991ec6e604daf9d9a77c3df9a6dac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.692986 8081 tablet_replica.cc:439] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } } }
I20260619 22:26:04.693113 8081 tablet_replica.cc:442] T fdc5eff774a0433cb726d5dbef0cdb83 P 9f3f1a05285742e4bb4d4ff762b958c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:04.693128 8102 mvcc.cc:205] Tried to move back new op lower bound from 7298695023326388224 to 7298695023247941632. Current Snapshot: MvccSnapshot[applied={T|T < 7298695023326388224}]
I20260619 22:26:04.693986 8103 mvcc.cc:205] Tried to move back new op lower bound from 7298695023326388224 to 7298695023247941632. Current Snapshot: MvccSnapshot[applied={T|T < 7298695023326388224}]
I20260619 22:26:04.706823 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.702956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.5.176.194:35485)
0619 22:26:04.703094 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.703098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.703372 (+ 274us) server_negotiation.cc:407] Connection header received
0619 22:26:04.703531 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.703534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.703586 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.703669 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:04.703961 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.704555 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.705631 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.705858 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.706019 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.706361 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:04.706438 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:04.706519 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.706615 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.706672 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:04.718506 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.715015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:37817 (local address 127.5.176.194:35485)
0619 22:26:04.715134 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:04.715139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:04.715330 (+ 191us) server_negotiation.cc:407] Connection header received
0619 22:26:04.715480 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:04.715483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:04.715524 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:04.715603 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:04.715847 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.716343 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.717281 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.717565 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.717672 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.718001 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:04.718075 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:04.718172 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:04.718252 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 22:26:04.718312 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":41}
I20260619 22:26:04.718566 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:04.714938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:35485 (local address 127.5.176.194:37817)
0619 22:26:04.715243 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:04.715256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:04.715380 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:04.715609 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:04.715611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:04.715617 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:04.715742 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.715748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.716445 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.716451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.717155 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:04.717164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:04.717670 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:04.717673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:04.717808 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:04.717852 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:04.718173 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:04.718179 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:26:04.718244 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 22:26:04.718321 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":22,"thread_start_us":103,"threads_started":1}
I20260619 22:26:04.726289 7998 raft_consensus.cc:1261] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:04.726734 8114 consensus_queue.cc:1035] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:04.726799 7725 raft_consensus.cc:1261] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:04.727207 8114 consensus_queue.cc:1035] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:04.771039 7568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:04.771238 7568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:04.773777 7568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Sending DeleteTablet for 3 replicas of tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c
I20260619 22:26:04.774736 7985 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:04 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:57176
I20260619 22:26:04.774757 7701 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:04 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260619 22:26:04.775303 8119 tablet_replica.cc:330] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: stopping tablet replica
I20260619 22:26:04.775632 8119 raft_consensus.cc:2229] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:04.775718 7844 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:04 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 22:26:04.775861 8119 raft_consensus.cc:2258] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:04.775921 7568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 22:26:04.776173 8121 tablet_replica.cc:330] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: stopping tablet replica
W20260619 22:26:04.776324 7568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:04.776448 8121 raft_consensus.cc:2229] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:04.776770 8121 raft_consensus.cc:2258] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:04.777242 8119 ts_tablet_manager.cc:1905] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:04.777875 8121 ts_tablet_manager.cc:1905] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:04.780326 8119 ts_tablet_manager.cc:1918] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:04.780581 8120 tablet_replica.cc:330] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: stopping tablet replica
I20260619 22:26:04.780802 8121 ts_tablet_manager.cc:1918] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:04.780916 8121 log.cc:1196] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/51ee21cb5cba451eaf8ec6d4f4a42f3c
I20260619 22:26:04.780912 8120 raft_consensus.cc:2229] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:04.781164 8120 raft_consensus.cc:2258] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:04.781288 8121 ts_tablet_manager.cc:1939] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P e3c618f3e40b46d4a3988e5adf58f576: Deleting consensus metadata
I20260619 22:26:04.781277 8119 log.cc:1196] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/51ee21cb5cba451eaf8ec6d4f4a42f3c
I20260619 22:26:04.781589 8119 ts_tablet_manager.cc:1939] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting consensus metadata
I20260619 22:26:04.782699 7985 tablet_service.cc:1467] Processing CreateTablet for tablet da9a489123f045438e5d198a2cdd110e (DEFAULT_TABLE table=test-table [id=b8a81d934ffa42b095742c659a2e5742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:04.782938 8120 ts_tablet_manager.cc:1905] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:04.782986 7985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9a489123f045438e5d198a2cdd110e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.783433 7552 catalog_manager.cc:4841] TS e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485): tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c (table test-table [id=6f867cd90a234b34bd6acba3a093f29b]) successfully deleted
I20260619 22:26:04.783805 7844 tablet_service.cc:1467] Processing CreateTablet for tablet da9a489123f045438e5d198a2cdd110e (DEFAULT_TABLE table=test-table [id=b8a81d934ffa42b095742c659a2e5742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:04.784060 7844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9a489123f045438e5d198a2cdd110e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.785238 8075 tablet_bootstrap.cc:492] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Bootstrap starting.
I20260619 22:26:04.782194 7701 tablet_service.cc:1467] Processing CreateTablet for tablet da9a489123f045438e5d198a2cdd110e (DEFAULT_TABLE table=test-table [id=b8a81d934ffa42b095742c659a2e5742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:04.785668 7701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9a489123f045438e5d198a2cdd110e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:04.786223 8075 tablet_bootstrap.cc:654] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.786473 8076 tablet_bootstrap.cc:492] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Bootstrap starting.
I20260619 22:26:04.787388 8076 tablet_bootstrap.cc:654] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.788084 8074 tablet_bootstrap.cc:492] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Bootstrap starting.
I20260619 22:26:04.788226 8075 tablet_bootstrap.cc:492] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: No bootstrap required, opened a new log
I20260619 22:26:04.788297 8075 ts_tablet_manager.cc:1397] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 22:26:04.788691 8075 raft_consensus.cc:348] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.788825 8075 raft_consensus.cc:374] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.788858 8075 raft_consensus.cc:729] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1991ec6e604daf9d9a77c3df9a6dac, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.788972 8075 consensus_queue.cc:260] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.789070 8074 tablet_bootstrap.cc:654] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:04.789219 8076 tablet_bootstrap.cc:492] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: No bootstrap required, opened a new log
I20260619 22:26:04.789242 8075 ts_tablet_manager.cc:1428] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:04.789294 8076 ts_tablet_manager.cc:1397] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:04.789728 8076 raft_consensus.cc:348] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.789836 8076 raft_consensus.cc:374] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.789865 8076 raft_consensus.cc:729] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.789973 8076 consensus_queue.cc:260] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.790228 8076 ts_tablet_manager.cc:1428] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:04.790994 7553 catalog_manager.cc:4841] TS 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453): tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c (table test-table [id=6f867cd90a234b34bd6acba3a093f29b]) successfully deleted
I20260619 22:26:04.791998 8120 ts_tablet_manager.cc:1918] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:04.792115 8120 log.cc:1196] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/51ee21cb5cba451eaf8ec6d4f4a42f3c
I20260619 22:26:04.792469 8120 ts_tablet_manager.cc:1939] T 51ee21cb5cba451eaf8ec6d4f4a42f3c P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting consensus metadata
I20260619 22:26:04.792946 8074 tablet_bootstrap.cc:492] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: No bootstrap required, opened a new log
I20260619 22:26:04.793030 8074 ts_tablet_manager.cc:1397] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 22:26:04.793474 8074 raft_consensus.cc:348] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.793578 8074 raft_consensus.cc:374] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:04.793607 8074 raft_consensus.cc:729] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f1a05285742e4bb4d4ff762b958c3, State: Initialized, Role: FOLLOWER
I20260619 22:26:04.793730 8074 consensus_queue.cc:260] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.793860 7554 catalog_manager.cc:4841] TS 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799): tablet 51ee21cb5cba451eaf8ec6d4f4a42f3c (table test-table [id=6f867cd90a234b34bd6acba3a093f29b]) successfully deleted
I20260619 22:26:04.793993 8074 ts_tablet_manager.cc:1428] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:04.810266 8080 raft_consensus.cc:482] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:04.810434 8080 raft_consensus.cc:504] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.810938 8080 leader_election.cc:288] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.811378 7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9a489123f045438e5d198a2cdd110e" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" is_pre_election: true
I20260619 22:26:04.811460 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9a489123f045438e5d198a2cdd110e" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" is_pre_election: true
I20260619 22:26:04.811560 7724 raft_consensus.cc:2454] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:04.811594 7998 raft_consensus.cc:2454] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:04.812078 7798 leader_election.cc:302] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:04.812337 8080 raft_consensus.cc:2781] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:04.812393 8080 raft_consensus.cc:482] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:04.812446 8080 raft_consensus.cc:3037] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.813786 8080 raft_consensus.cc:504] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.814375 8080 leader_election.cc:288] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Requested vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:04.814559 7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9a489123f045438e5d198a2cdd110e" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
I20260619 22:26:04.814703 7724 raft_consensus.cc:3037] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.814797 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9a489123f045438e5d198a2cdd110e" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
I20260619 22:26:04.814926 7998 raft_consensus.cc:3037] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:04.815980 7724 raft_consensus.cc:2454] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:04.815980 7998 raft_consensus.cc:2454] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:04.816354 7798 leader_election.cc:302] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:04.816637 8080 raft_consensus.cc:2781] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:04.816735 8080 raft_consensus.cc:686] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Becoming Leader. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Running, Role: LEADER
I20260619 22:26:04.816875 8080 consensus_queue.cc:237] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:04.818764 7567 catalog_manager.cc:5495] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:04.844473 7998 raft_consensus.cc:1261] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:04.844892 8080 consensus_queue.cc:1035] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:04.845237 7724 raft_consensus.cc:1261] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:04.845661 8080 consensus_queue.cc:1035] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:26:04.860937 8101 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 22:26:06.419054 7568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:06.419210 7568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:06.421614 7568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Sending DeleteTablet for 3 replicas of tablet da9a489123f045438e5d198a2cdd110e
I20260619 22:26:06.422127 7844 tablet_service.cc:1514] Processing DeleteTablet for tablet da9a489123f045438e5d198a2cdd110e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:06 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 22:26:06.422307 7701 tablet_service.cc:1514] Processing DeleteTablet for tablet da9a489123f045438e5d198a2cdd110e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:06 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260619 22:26:06.422535 8147 tablet_replica.cc:330] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: stopping tablet replica
I20260619 22:26:06.422677 8148 tablet_replica.cc:330] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: stopping tablet replica
I20260619 22:26:06.422693 8147 raft_consensus.cc:2229] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:06.422747 7985 tablet_service.cc:1514] Processing DeleteTablet for tablet da9a489123f045438e5d198a2cdd110e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:06 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:57176
I20260619 22:26:06.422871 8148 raft_consensus.cc:2229] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:06.422899 8147 raft_consensus.cc:2258] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:06.422991 8148 raft_consensus.cc:2258] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:06.423159 8149 tablet_replica.cc:330] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: stopping tablet replica
I20260619 22:26:06.423341 8149 raft_consensus.cc:2229] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:06.423468 8149 raft_consensus.cc:2258] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:06.423734 8148 ts_tablet_manager.cc:1905] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:06.424150 8149 ts_tablet_manager.cc:1905] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:06.426033 8147 ts_tablet_manager.cc:1905] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:06.426645 8148 ts_tablet_manager.cc:1918] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:26:06.426744 8148 log.cc:1196] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/da9a489123f045438e5d198a2cdd110e
I20260619 22:26:06.426905 8149 ts_tablet_manager.cc:1918] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:26:06.426980 8149 log.cc:1196] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/da9a489123f045438e5d198a2cdd110e
I20260619 22:26:06.427066 8148 ts_tablet_manager.cc:1939] T da9a489123f045438e5d198a2cdd110e P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting consensus metadata
I20260619 22:26:06.427273 8149 ts_tablet_manager.cc:1939] T da9a489123f045438e5d198a2cdd110e P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting consensus metadata
I20260619 22:26:06.427780 7553 catalog_manager.cc:4841] TS 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453): tablet da9a489123f045438e5d198a2cdd110e (table test-table [id=b8a81d934ffa42b095742c659a2e5742]) successfully deleted
I20260619 22:26:06.427917 7554 catalog_manager.cc:4841] TS 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799): tablet da9a489123f045438e5d198a2cdd110e (table test-table [id=b8a81d934ffa42b095742c659a2e5742]) successfully deleted
I20260619 22:26:06.429059 8147 ts_tablet_manager.cc:1918] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:26:06.429142 8147 log.cc:1196] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/da9a489123f045438e5d198a2cdd110e
I20260619 22:26:06.429448 8147 ts_tablet_manager.cc:1939] T da9a489123f045438e5d198a2cdd110e P e3c618f3e40b46d4a3988e5adf58f576: Deleting consensus metadata
I20260619 22:26:06.430584 7552 catalog_manager.cc:4841] TS e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485): tablet da9a489123f045438e5d198a2cdd110e (table test-table [id=b8a81d934ffa42b095742c659a2e5742]) successfully deleted
Jun 19 22:26:06 dist-test-slave-bt27 krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907966, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:26:06 dist-test-slave-bt27 krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907966, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:06.451570 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.442527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.5.176.254:43361)
0619 22:26:06.442855 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.442866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.442968 (+ 102us) server_negotiation.cc:407] Connection header received
0619 22:26:06.443152 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.443155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.443199 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.443302 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:06.444027 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.444603 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.445465 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.445699 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.445812 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.448383 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:06.448386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:06.448388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:06.448411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:06.449726 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:06.450256 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:06.450259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:06.450266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:06.450347 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:06.450652 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:06.450655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:06.450659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:06.450855 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:06.451006 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.451244 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.451378 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20260619 22:26:06.454311 7568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:06.454758 7568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:06.459422 7701 tablet_service.cc:1467] Processing CreateTablet for tablet 17a92d84e36849a7ba33c97e6157cd51 (DEFAULT_TABLE table=test-table [id=0524d8612c064be3a0b095f5a511251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:06.459425 7844 tablet_service.cc:1467] Processing CreateTablet for tablet 17a92d84e36849a7ba33c97e6157cd51 (DEFAULT_TABLE table=test-table [id=0524d8612c064be3a0b095f5a511251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:06.459609 7985 tablet_service.cc:1467] Processing CreateTablet for tablet 17a92d84e36849a7ba33c97e6157cd51 (DEFAULT_TABLE table=test-table [id=0524d8612c064be3a0b095f5a511251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:06.459743 7844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a92d84e36849a7ba33c97e6157cd51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:06.459753 7701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a92d84e36849a7ba33c97e6157cd51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:06.459868 7985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a92d84e36849a7ba33c97e6157cd51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:06.461778 8158 tablet_bootstrap.cc:492] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Bootstrap starting.
I20260619 22:26:06.462525 8160 tablet_bootstrap.cc:492] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Bootstrap starting.
I20260619 22:26:06.462836 8158 tablet_bootstrap.cc:654] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:06.463313 8160 tablet_bootstrap.cc:654] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:06.463943 8159 tablet_bootstrap.cc:492] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Bootstrap starting.
I20260619 22:26:06.464499 8158 tablet_bootstrap.cc:492] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: No bootstrap required, opened a new log
I20260619 22:26:06.464541 8160 tablet_bootstrap.cc:492] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: No bootstrap required, opened a new log
I20260619 22:26:06.464636 8160 ts_tablet_manager.cc:1397] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:06.464883 8159 tablet_bootstrap.cc:654] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:06.464670 8158 ts_tablet_manager.cc:1397] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:06.465301 8160 raft_consensus.cc:348] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.465447 8160 raft_consensus.cc:374] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:06.465489 8160 raft_consensus.cc:729] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1991ec6e604daf9d9a77c3df9a6dac, State: Initialized, Role: FOLLOWER
I20260619 22:26:06.465535 8158 raft_consensus.cc:348] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.465659 8158 raft_consensus.cc:374] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:06.465636 8160 consensus_queue.cc:260] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.465703 8158 raft_consensus.cc:729] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f1a05285742e4bb4d4ff762b958c3, State: Initialized, Role: FOLLOWER
I20260619 22:26:06.465852 8158 consensus_queue.cc:260] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.466027 8160 ts_tablet_manager.cc:1428] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:26:06.466286 8159 tablet_bootstrap.cc:492] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: No bootstrap required, opened a new log
I20260619 22:26:06.466292 8158 ts_tablet_manager.cc:1428] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:06.466375 8159 ts_tablet_manager.cc:1397] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:06.466768 8159 raft_consensus.cc:348] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.466892 8159 raft_consensus.cc:374] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:06.466925 8159 raft_consensus.cc:729] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Initialized, Role: FOLLOWER
I20260619 22:26:06.467087 8159 consensus_queue.cc:260] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.468078 8159 ts_tablet_manager.cc:1428] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:06.584554 8164 raft_consensus.cc:482] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:06.584745 8164 raft_consensus.cc:504] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.585949 8164 leader_election.cc:288] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485)
I20260619 22:26:06.590270 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.586174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:35485 (local address 127.5.176.195:35719)
0619 22:26:06.586519 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:06.586536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:06.586642 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:06.586985 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:06.586988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:06.587011 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:06.587115 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.587119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.587998 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.588001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.588984 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.588992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.589782 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.589786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.589885 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.589912 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:26:06.589997 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 22:26:06.590073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":126,"threads_started":1}
I20260619 22:26:06.590353 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.586276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:35719 (local address 127.5.176.194:35485)
0619 22:26:06.586609 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.586618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.586632 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:06.586767 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.586770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.586868 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.586957 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:06.587215 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.587881 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.589111 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.589643 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.589732 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.589772 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.590060 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.590206 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":147,"threads_started":1}
I20260619 22:26:06.590884 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.586290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:54561 (local address 127.5.176.193:39453)
0619 22:26:06.586595 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.586604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.586843 (+ 239us) server_negotiation.cc:407] Connection header received
0619 22:26:06.587240 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.587242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.587281 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.587380 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:06.587643 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.588343 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.589467 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.590054 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.590203 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.590245 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.590705 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.590769 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
I20260619 22:26:06.590799 7865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a92d84e36849a7ba33c97e6157cd51" candidate_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c618f3e40b46d4a3988e5adf58f576" is_pre_election: true
I20260619 22:26:06.590904 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.586179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39453 (local address 127.5.176.195:54561)
0619 22:26:06.586746 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:06.586757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:06.586882 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:06.587397 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:06.587400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:06.587416 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:06.587530 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.587535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.588461 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.588463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.589334 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.589344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.590217 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.590221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.590322 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.590346 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:26:06.590690 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 22:26:06.590760 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":270,"thread_start_us":80,"threads_started":1}
I20260619 22:26:06.591073 7865 raft_consensus.cc:2454] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1991ec6e604daf9d9a77c3df9a6dac in term 0.
I20260619 22:26:06.591503 7936 leader_election.cc:302] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:06.591542 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a92d84e36849a7ba33c97e6157cd51" candidate_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" is_pre_election: true
I20260619 22:26:06.591665 7725 raft_consensus.cc:2454] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1991ec6e604daf9d9a77c3df9a6dac in term 0.
I20260619 22:26:06.591773 8164 raft_consensus.cc:2781] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:06.591830 8164 raft_consensus.cc:482] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:06.591878 8164 raft_consensus.cc:3037] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:06.592811 8164 raft_consensus.cc:504] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.593122 8164 leader_election.cc:288] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [CANDIDATE]: Term 1 election: Requested vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485)
I20260619 22:26:06.593426 7865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a92d84e36849a7ba33c97e6157cd51" candidate_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c618f3e40b46d4a3988e5adf58f576"
I20260619 22:26:06.593552 7865 raft_consensus.cc:3037] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:06.593511 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a92d84e36849a7ba33c97e6157cd51" candidate_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
I20260619 22:26:06.593645 7725 raft_consensus.cc:3037] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:06.594720 7725 raft_consensus.cc:2454] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1991ec6e604daf9d9a77c3df9a6dac in term 1.
I20260619 22:26:06.594723 7865 raft_consensus.cc:2454] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1991ec6e604daf9d9a77c3df9a6dac in term 1.
I20260619 22:26:06.595016 7937 leader_election.cc:302] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, 9f3f1a05285742e4bb4d4ff762b958c3; no voters:
I20260619 22:26:06.595201 8164 raft_consensus.cc:2781] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:06.595425 8164 raft_consensus.cc:686] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 LEADER]: Becoming Leader. State: Replica: 3d1991ec6e604daf9d9a77c3df9a6dac, State: Running, Role: LEADER
I20260619 22:26:06.595691 8164 consensus_queue.cc:237] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.597883 7567 catalog_manager.cc:5495] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195). New cstate: current_term: 1 leader_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } health_report { overall_health: HEALTHY } } }
I20260619 22:26:06.628068 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.624379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.5.176.195:43799)
0619 22:26:06.624658 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.624665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.624676 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:26:06.624751 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.624753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.624802 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.624907 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:06.625185 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.625728 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.626571 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.626799 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.626874 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.627111 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:06.627573 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:06.627660 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.627818 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.627860 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":132,"threads_started":1}
I20260619 22:26:06.639150 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.634400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:35257 (local address 127.5.176.194:35485)
0619 22:26:06.634525 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.634530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.634648 (+ 118us) server_negotiation.cc:407] Connection header received
0619 22:26:06.634729 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.634732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.634791 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.634877 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:06.635088 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.635857 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.637357 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.637863 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.637960 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.638504 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:06.638608 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:06.638661 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.638877 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.638958 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:26:06.639174 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.634285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:35485 (local address 127.5.176.195:35257)
0619 22:26:06.634567 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:06.634580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:06.634661 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:06.634896 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:06.634899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:06.634904 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:06.634993 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.634997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.636009 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.636014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.637224 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.637238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.638096 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.638100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.638246 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.638309 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:06.638733 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:06.638742 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:26:06.638819 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 22:26:06.638922 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":122,"threads_started":1}
I20260619 22:26:06.651948 7725 raft_consensus.cc:1261] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Refusing update from remote peer 3d1991ec6e604daf9d9a77c3df9a6dac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:06.652015 7865 raft_consensus.cc:1261] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Refusing update from remote peer 3d1991ec6e604daf9d9a77c3df9a6dac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:06.652478 8170 consensus_queue.cc:1035] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:06.652707 8164 consensus_queue.cc:1035] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:06.672093 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.667898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:42911 (local address 127.5.176.195:43799)
0619 22:26:06.668004 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.668009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.668020 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:26:06.668064 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.668066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.668102 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.668189 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:06.668448 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.669430 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.670553 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.670992 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.671125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.671618 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:06.671689 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:06.671748 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.671940 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.671989 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 22:26:06.672209 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.667475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:43799 (local address 127.5.176.194:42911)
0619 22:26:06.667769 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:06.667789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:06.667892 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:06.668198 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:06.668201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:06.668208 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:06.668313 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.668319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.669554 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.669558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.670434 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:06.670444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.671232 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.671238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:06.671449 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.671498 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:06.671784 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:06.671792 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:26:06.671871 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 22:26:06.671983 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
I20260619 22:26:06.699430 7565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:06.699644 7565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:06.702577 7565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Sending DeleteTablet for 3 replicas of tablet 17a92d84e36849a7ba33c97e6157cd51
I20260619 22:26:06.703284 7701 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a92d84e36849a7ba33c97e6157cd51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:06 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260619 22:26:06.703474 7985 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a92d84e36849a7ba33c97e6157cd51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:06 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:57176
I20260619 22:26:06.703543 8148 tablet_replica.cc:330] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: stopping tablet replica
I20260619 22:26:06.703670 8148 raft_consensus.cc:2229] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:06.703681 8149 tablet_replica.cc:330] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: stopping tablet replica
I20260619 22:26:06.703799 8148 raft_consensus.cc:2258] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:06.703837 8149 raft_consensus.cc:2229] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:06.704209 8149 raft_consensus.cc:2258] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:06.704504 7565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:06.704955 7565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:06.705497 8148 ts_tablet_manager.cc:1905] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:06.706918 8149 ts_tablet_manager.cc:1905] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:06.709786 7985 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0595d76501490a8cb82c212b786761 (DEFAULT_TABLE table=test-table [id=157b04b78c314373bebc1a3a6ec11bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:06.710129 7985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0595d76501490a8cb82c212b786761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:06.712566 7701 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0595d76501490a8cb82c212b786761 (DEFAULT_TABLE table=test-table [id=157b04b78c314373bebc1a3a6ec11bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:06.712806 7701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0595d76501490a8cb82c212b786761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:06.713213 7844 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a92d84e36849a7ba33c97e6157cd51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:06 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 22:26:06.713442 8147 tablet_replica.cc:330] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: stopping tablet replica
I20260619 22:26:06.713585 8147 raft_consensus.cc:2229] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:06.713716 8147 raft_consensus.cc:2258] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:06.714659 8147 ts_tablet_manager.cc:1905] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:06.714906 8158 tablet_bootstrap.cc:492] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3: Bootstrap starting.
I20260619 22:26:06.715832 8158 tablet_bootstrap.cc:654] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:06.716897 8160 tablet_bootstrap.cc:492] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Bootstrap starting.
I20260619 22:26:06.717675 7845 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0595d76501490a8cb82c212b786761 (DEFAULT_TABLE table=test-table [id=157b04b78c314373bebc1a3a6ec11bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:06.717789 8160 tablet_bootstrap.cc:654] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:06.717960 7845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0595d76501490a8cb82c212b786761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:06.720355 8158 tablet_bootstrap.cc:492] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3: No bootstrap required, opened a new log
I20260619 22:26:06.720443 8158 ts_tablet_manager.cc:1397] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 22:26:06.720882 8159 tablet_bootstrap.cc:492] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Bootstrap starting.
I20260619 22:26:06.720885 8158 raft_consensus.cc:348] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.721015 8158 raft_consensus.cc:374] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:06.721048 8158 raft_consensus.cc:729] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f1a05285742e4bb4d4ff762b958c3, State: Initialized, Role: FOLLOWER
I20260619 22:26:06.721163 8158 consensus_queue.cc:260] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.721432 8158 ts_tablet_manager.cc:1428] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:06.722011 8159 tablet_bootstrap.cc:654] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:06.723774 8160 tablet_bootstrap.cc:492] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: No bootstrap required, opened a new log
I20260619 22:26:06.723868 8160 ts_tablet_manager.cc:1397] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 22:26:06.724303 8160 raft_consensus.cc:348] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.724431 8160 raft_consensus.cc:374] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:06.724501 8160 raft_consensus.cc:729] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1991ec6e604daf9d9a77c3df9a6dac, State: Initialized, Role: FOLLOWER
I20260619 22:26:06.724643 8160 consensus_queue.cc:260] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.724787 8147 ts_tablet_manager.cc:1918] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:06.724903 8160 ts_tablet_manager.cc:1428] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:06.724907 8147 log.cc:1196] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/17a92d84e36849a7ba33c97e6157cd51
I20260619 22:26:06.725015 8148 ts_tablet_manager.cc:1918] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:06.725170 8149 ts_tablet_manager.cc:1918] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:06.725565 8147 ts_tablet_manager.cc:1939] T 17a92d84e36849a7ba33c97e6157cd51 P e3c618f3e40b46d4a3988e5adf58f576: Deleting consensus metadata
I20260619 22:26:06.725675 8148 log.cc:1196] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/17a92d84e36849a7ba33c97e6157cd51
I20260619 22:26:06.725795 8149 log.cc:1196] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/17a92d84e36849a7ba33c97e6157cd51
I20260619 22:26:06.726122 8148 ts_tablet_manager.cc:1939] T 17a92d84e36849a7ba33c97e6157cd51 P 9f3f1a05285742e4bb4d4ff762b958c3: Deleting consensus metadata
I20260619 22:26:06.727123 8149 ts_tablet_manager.cc:1939] T 17a92d84e36849a7ba33c97e6157cd51 P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting consensus metadata
I20260619 22:26:06.726758 8159 tablet_bootstrap.cc:492] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: No bootstrap required, opened a new log
I20260619 22:26:06.727285 8159 ts_tablet_manager.cc:1397] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260619 22:26:06.727808 8159 raft_consensus.cc:348] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.727938 8159 raft_consensus.cc:374] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:06.727977 8159 raft_consensus.cc:729] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Initialized, Role: FOLLOWER
I20260619 22:26:06.728289 7553 catalog_manager.cc:4841] TS 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453): tablet 17a92d84e36849a7ba33c97e6157cd51 (table test-table [id=0524d8612c064be3a0b095f5a511251e]) successfully deleted
I20260619 22:26:06.728251 8159 consensus_queue.cc:260] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.729295 8159 ts_tablet_manager.cc:1428] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 22:26:06.729522 7552 catalog_manager.cc:4841] TS e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485): tablet 17a92d84e36849a7ba33c97e6157cd51 (table test-table [id=0524d8612c064be3a0b095f5a511251e]) successfully deleted
I20260619 22:26:06.730170 7554 catalog_manager.cc:4841] TS 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799): tablet 17a92d84e36849a7ba33c97e6157cd51 (table test-table [id=0524d8612c064be3a0b095f5a511251e]) successfully deleted
I20260619 22:26:06.902887 8130 raft_consensus.cc:482] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:06.903080 8130 raft_consensus.cc:504] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.903580 8130 leader_election.cc:288] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:06.904104 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0595d76501490a8cb82c212b786761" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" is_pre_election: true
I20260619 22:26:06.904155 8005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0595d76501490a8cb82c212b786761" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" is_pre_election: true
I20260619 22:26:06.904284 7725 raft_consensus.cc:2454] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:06.904304 8005 raft_consensus.cc:2454] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 0.
I20260619 22:26:06.904667 7797 leader_election.cc:302] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3f1a05285742e4bb4d4ff762b958c3, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:06.904927 8130 raft_consensus.cc:2781] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:06.904983 8130 raft_consensus.cc:482] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:06.905042 8130 raft_consensus.cc:3037] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:06.906059 8130 raft_consensus.cc:504] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.906416 8130 leader_election.cc:288] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Requested vote from peers 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453), 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799)
I20260619 22:26:06.906803 7725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0595d76501490a8cb82c212b786761" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3f1a05285742e4bb4d4ff762b958c3"
I20260619 22:26:06.906803 8005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0595d76501490a8cb82c212b786761" candidate_uuid: "e3c618f3e40b46d4a3988e5adf58f576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac"
I20260619 22:26:06.906939 7725 raft_consensus.cc:3037] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:06.906939 8005 raft_consensus.cc:3037] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:06.908118 8005 raft_consensus.cc:2454] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:06.908118 7725 raft_consensus.cc:2454] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c618f3e40b46d4a3988e5adf58f576 in term 1.
I20260619 22:26:06.908413 7798 leader_election.cc:302] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1991ec6e604daf9d9a77c3df9a6dac, e3c618f3e40b46d4a3988e5adf58f576; no voters:
I20260619 22:26:06.908618 8130 raft_consensus.cc:2781] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:06.908776 8130 raft_consensus.cc:686] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Becoming Leader. State: Replica: e3c618f3e40b46d4a3988e5adf58f576, State: Running, Role: LEADER
I20260619 22:26:06.908936 8130 consensus_queue.cc:237] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } }
I20260619 22:26:06.910661 7565 catalog_manager.cc:5495] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "e3c618f3e40b46d4a3988e5adf58f576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c618f3e40b46d4a3988e5adf58f576" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:06.925215 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:06.922054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.5.176.194:35485)
0619 22:26:06.922215 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:06.922221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:06.922234 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:06.922389 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:06.922392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:06.922446 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:06.922524 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:06.922771 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.923299 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.924160 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:06.924400 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:06.924521 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:06.924768 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:06.924849 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:06.924929 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:06.925024 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 22:26:06.925076 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:26:06.931294 8005 raft_consensus.cc:1261] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:06.931397 7725 raft_consensus.cc:1261] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Refusing update from remote peer e3c618f3e40b46d4a3988e5adf58f576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:06.931690 8130 consensus_queue.cc:1035] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1991ec6e604daf9d9a77c3df9a6dac" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:06.931953 8136 consensus_queue.cc:1035] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3f1a05285742e4bb4d4ff762b958c3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:26:06.944978 8180 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 22:26:08.497953 7565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:08.498139 7565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:08.500821 7565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3ab4accd0464fd3ab61e65f2da9e914: Sending DeleteTablet for 3 replicas of tablet 1a0595d76501490a8cb82c212b786761
I20260619 22:26:08.501434 7701 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a0595d76501490a8cb82c212b786761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:08 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260619 22:26:08.501641 7985 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a0595d76501490a8cb82c212b786761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:08 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:57176
I20260619 22:26:08.501809 8210 tablet_replica.cc:330] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3: stopping tablet replica
I20260619 22:26:08.501942 7844 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a0595d76501490a8cb82c212b786761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:08 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 22:26:08.501947 8210 raft_consensus.cc:2229] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:08.502094 8210 raft_consensus.cc:2258] T 1a0595d76501490a8cb82c212b786761 P 9f3f1a05285742e4bb4d4ff762b958c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:08.502146 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7640
I20260619 22:26:08.504117 8212 tablet_replica.cc:330] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: stopping tablet replica
I20260619 22:26:08.504249 8212 raft_consensus.cc:2229] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:08.504423 8212 raft_consensus.cc:2258] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:08.504688 8211 tablet_replica.cc:330] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: stopping tablet replica
I20260619 22:26:08.504822 8211 raft_consensus.cc:2229] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:08.505000 8211 raft_consensus.cc:2258] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:08.505227 8212 ts_tablet_manager.cc:1905] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:08.505661 8211 ts_tablet_manager.cc:1905] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:08.508055 8212 ts_tablet_manager.cc:1918] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:26:08.508147 8212 log.cc:1196] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/1a0595d76501490a8cb82c212b786761
I20260619 22:26:08.508396 8211 ts_tablet_manager.cc:1918] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 22:26:08.508462 8212 ts_tablet_manager.cc:1939] T 1a0595d76501490a8cb82c212b786761 P e3c618f3e40b46d4a3988e5adf58f576: Deleting consensus metadata
I20260619 22:26:08.508482 8211 log.cc:1196] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/1a0595d76501490a8cb82c212b786761
I20260619 22:26:08.508713 8211 ts_tablet_manager.cc:1939] T 1a0595d76501490a8cb82c212b786761 P 3d1991ec6e604daf9d9a77c3df9a6dac: Deleting consensus metadata
I20260619 22:26:08.509315 7552 catalog_manager.cc:4841] TS e3c618f3e40b46d4a3988e5adf58f576 (127.5.176.194:35485): tablet 1a0595d76501490a8cb82c212b786761 (table test-table [id=157b04b78c314373bebc1a3a6ec11bad]) successfully deleted
I20260619 22:26:08.509443 7554 catalog_manager.cc:4841] TS 3d1991ec6e604daf9d9a77c3df9a6dac (127.5.176.195:43799): tablet 1a0595d76501490a8cb82c212b786761 (table test-table [id=157b04b78c314373bebc1a3a6ec11bad]) successfully deleted
W20260619 22:26:08.511214 7553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:39453 (error 108)
I20260619 22:26:08.511480 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7780
I20260619 22:26:08.513295 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:08.512831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39453 (local address 127.0.0.1:55314)
0619 22:26:08.513110 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:08.513190 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:39453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
W20260619 22:26:08.513463 7553 catalog_manager.cc:4584] TS 9f3f1a05285742e4bb4d4ff762b958c3 (127.5.176.193:39453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a0595d76501490a8cb82c212b786761: Network error: Client connection negotiation failed: client connection to 127.5.176.193:39453: connect: Connection refused (error 111)
I20260619 22:26:08.521229 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7920
I20260619 22:26:08.528949 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 7536
2026-06-19T22:26:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6711 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:08 dist-test-slave-bt27 krb5kdc[8217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:08 dist-test-slave-bt27 krb5kdc[8217](info): set up 2 sockets
Jun 19 22:26:08 dist-test-slave-bt27 krb5kdc[8217](info): commencing operation
krb5kdc: starting...
W20260619 22:26:09.586319 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:09 dist-test-slave-bt27 krb5kdc[8217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:09.707353 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43273
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:43273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:09.823107 8230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:09.823459 8230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:09.823534 8230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:09.827736 8230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:09.827826 8230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:09.827864 8230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:09.827932 8230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:09.827983 8230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:09.833493 8230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:43273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:09.834769 8230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:09.835863 8230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:26:09.845208 8230 server_base.cc:1034] running on GCE node
W20260619 22:26:09.845157 8238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:09.845191 8235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:09.845157 8236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:09.846006 8230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:09.847203 8230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:09.848366 8230 hybrid_clock.cc:648] HybridClock initialized: now 1781907969848353 us; error 26 us; skew 500 ppm
Jun 19 22:26:09 dist-test-slave-bt27 krb5kdc[8217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:09.851239 8230 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:09.852228 8230 webserver.cc:460] Webserver started at http://127.5.176.254:42335/ using document root <none> and password file <none>
I20260619 22:26:09.852766 8230 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:09.852828 8230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:09.853008 8230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:09.855264 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "2046861da6e04124b8f493c0419cb6f2"
format_stamp: "Formatted at 2026-06-19 22:26:09 on dist-test-slave-bt27"
server_key: "803a454befd641f3ae8285ac80ffac72"
server_key_iv: "c50220b95d5b9974c581ae25f4f12447"
server_key_version: "encryptionkey@0"
I20260619 22:26:09.855814 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "2046861da6e04124b8f493c0419cb6f2"
format_stamp: "Formatted at 2026-06-19 22:26:09 on dist-test-slave-bt27"
server_key: "803a454befd641f3ae8285ac80ffac72"
server_key_iv: "c50220b95d5b9974c581ae25f4f12447"
server_key_version: "encryptionkey@0"
I20260619 22:26:09.859681 8230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:09.862340 8245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:09.863709 8230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:26:09.863844 8230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "2046861da6e04124b8f493c0419cb6f2"
format_stamp: "Formatted at 2026-06-19 22:26:09 on dist-test-slave-bt27"
server_key: "803a454befd641f3ae8285ac80ffac72"
server_key_iv: "c50220b95d5b9974c581ae25f4f12447"
server_key_version: "encryptionkey@0"
I20260619 22:26:09.863970 8230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:09.879999 8230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:09.883272 8230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:09.883479 8230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:09.890812 8230 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:43273
I20260619 22:26:09.890820 8297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:43273 every 8 connection(s)
I20260619 22:26:09.891739 8230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:09.893983 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8230
I20260619 22:26:09.894073 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:09.894378 5827 external_mini_cluster.cc:1428] Setting key aa106f61c5fc6bd984a8af86aad58658
I20260619 22:26:09.894596 8298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:09.900738 8298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2: Bootstrap starting.
I20260619 22:26:09.903030 8298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2: Neither blocks nor log segments found. Creating new log.
Jun 19 22:26:09 dist-test-slave-bt27 krb5kdc[8217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:09.903926 8298 log.cc:824] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:09.905920 8298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2: No bootstrap required, opened a new log
I20260619 22:26:09.909014 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:09.895803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.5.176.254:43273)
0619 22:26:09.896262 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:09.896277 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:09.896314 (+ 37us) server_negotiation.cc:407] Connection header received
0619 22:26:09.897212 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:09.897231 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:09.897551 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:09.897884 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:09.898713 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:09.899709 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:09.900567 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:09.901099 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:09.901280 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:09.904096 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:09.904101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:09.904113 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:09.904140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:09.906210 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:09.906716 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:09.906720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:09.906725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:09.906832 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:09.907125 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:09.907127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:09.907128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:09.907509 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:09.907654 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:09.907970 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 22:26:09.908264 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":137,"threads_started":1}
I20260619 22:26:09.910068 8298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2046861da6e04124b8f493c0419cb6f2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43273 } }
I20260619 22:26:09.910368 8298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:09.910419 8298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2046861da6e04124b8f493c0419cb6f2, State: Initialized, Role: FOLLOWER
I20260619 22:26:09.910883 8298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2046861da6e04124b8f493c0419cb6f2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43273 } }
I20260619 22:26:09.911036 8298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:09.911101 8298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:09.911206 8298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:09.912161 8298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2046861da6e04124b8f493c0419cb6f2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43273 } }
I20260619 22:26:09.912504 8298 leader_election.cc:302] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2046861da6e04124b8f493c0419cb6f2; no voters:
I20260619 22:26:09.912822 8298 leader_election.cc:288] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:09.912948 8303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:09.913205 8303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2046861da6e04124b8f493c0419cb6f2, State: Running, Role: LEADER
I20260619 22:26:09.913516 8303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2046861da6e04124b8f493c0419cb6f2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43273 } }
I20260619 22:26:09.914000 8298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:09.915144 8304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2046861da6e04124b8f493c0419cb6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2046861da6e04124b8f493c0419cb6f2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43273 } } }
I20260619 22:26:09.915280 8304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:09.915642 8305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2046861da6e04124b8f493c0419cb6f2. Latest consensus state: current_term: 1 leader_uuid: "2046861da6e04124b8f493c0419cb6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2046861da6e04124b8f493c0419cb6f2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43273 } } }
I20260619 22:26:09.915760 8305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:09.915699 8312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:09.918777 8312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:09.924536 8312 catalog_manager.cc:1347] Generated new cluster ID: 04167907876943588d4224559961fb7d
I20260619 22:26:09.924619 8312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:09.945165 8312 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:09.946100 8312 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:09.955950 8312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2046861da6e04124b8f493c0419cb6f2: Generated new TSK 0
I20260619 22:26:09.956698 8312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:09.988807 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43273
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:10.124769 8324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:10.125080 8324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:10.125129 8324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:10.129670 8324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:10.129751 8324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:10.129815 8324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:10.135613 8324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:10.136837 8324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:10.137913 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:10.148694 8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:10.148840 8332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:10.148747 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:10.149061 8324 server_base.cc:1034] running on GCE node
I20260619 22:26:10.149552 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:10.150379 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:10.151568 8324 hybrid_clock.cc:648] HybridClock initialized: now 1781907970151548 us; error 35 us; skew 500 ppm
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:10.154907 8324 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:10.155905 8324 webserver.cc:460] Webserver started at http://127.5.176.193:40091/ using document root <none> and password file <none>
I20260619 22:26:10.156555 8324 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:10.156628 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:10.156818 8324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:10.159093 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "20c33950ae3d42c0b47f6c2db9c75a3e"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "d98307299b36e71c53740aae67c4963b"
server_key_iv: "64f499558db17a2f85e51f23d250953f"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.159662 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "20c33950ae3d42c0b47f6c2db9c75a3e"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "d98307299b36e71c53740aae67c4963b"
server_key_iv: "64f499558db17a2f85e51f23d250953f"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.163559 8324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:10.166322 8339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.167786 8324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:10.167888 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "20c33950ae3d42c0b47f6c2db9c75a3e"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "d98307299b36e71c53740aae67c4963b"
server_key_iv: "64f499558db17a2f85e51f23d250953f"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.167986 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:10.194482 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:10.197834 8324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:10.198010 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:10.198691 8324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:10.199554 8324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:10.199611 8324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.199654 8324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:10.199674 8324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.209409 8324 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:35861
I20260619 22:26:10.209429 8452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:35861 every 8 connection(s)
I20260619 22:26:10.210706 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:10.217727 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8324
I20260619 22:26:10.217876 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:10.218187 5827 external_mini_cluster.cc:1428] Setting key f3a92d03b11ccd36795e20844deebc11
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:10.225965 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:10.212819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:51651 (local address 127.5.176.254:43273)
0619 22:26:10.212982 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:10.212986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:10.213839 (+ 853us) server_negotiation.cc:407] Connection header received
0619 22:26:10.215346 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:10.215350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:10.215427 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:10.215510 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:10.216743 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.217287 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.218691 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.218942 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.219058 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:10.221768 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:10.221774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:10.221779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:10.221813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:10.223674 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:10.224216 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:10.224220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:10.224221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:10.224317 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:10.224685 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:10.224689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:10.224691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:10.224977 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:10.225095 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:10.225612 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 22:26:10.225768 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:26:10.226775 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:10.213223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43273 (local address 127.5.176.193:51651)
0619 22:26:10.213698 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:10.213740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:10.215138 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:10.215699 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:10.215707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:10.216113 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:10.216590 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:10.216603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.217445 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.217448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:10.218568 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:10.218577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.219096 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.219100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:10.219279 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:10.219872 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:10.219903 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:10.221583 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:10.223808 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:10.223813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:10.223834 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:10.224104 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:10.224424 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:10.224427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:10.224428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:10.224582 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:10.225087 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:10.225092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:10.225363 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 22:26:10.225527 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 22:26:10.225806 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:26:10.228132 8453 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43273
I20260619 22:26:10.228402 8453 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:10.228901 8453 heartbeater.cc:511] Master 127.5.176.254:43273 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:10.230682 8262 ts_manager.cc:194] Registered new tserver with Master: 20c33950ae3d42c0b47f6c2db9c75a3e (127.5.176.193:35861)
I20260619 22:26:10.231840 8262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:51651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:10.245289 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43273
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:10.358006 8458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:10.358386 8458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:10.358474 8458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:10.362545 8458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:10.362650 8458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:10.362746 8458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:10.368348 8458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:10.369493 8458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:10.370543 8458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:10.380172 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:10.380183 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:10.380190 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:10.380239 8458 server_base.cc:1034] running on GCE node
I20260619 22:26:10.380841 8458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:10.381510 8458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:10.382694 8458 hybrid_clock.cc:648] HybridClock initialized: now 1781907970382678 us; error 29 us; skew 500 ppm
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:10.385524 8458 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:10.386518 8458 webserver.cc:460] Webserver started at http://127.5.176.194:41025/ using document root <none> and password file <none>
I20260619 22:26:10.387104 8458 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:10.387192 8458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:10.387423 8458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:10.389423 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "8a5893db6eb540d1b7075b4038b32569"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "f4d4e2b858854bebf137f739d584653b"
server_key_iv: "767c8f407a84dd95ec4b087ca8fa1928"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.389981 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "8a5893db6eb540d1b7075b4038b32569"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "f4d4e2b858854bebf137f739d584653b"
server_key_iv: "767c8f407a84dd95ec4b087ca8fa1928"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.393692 8458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:10.396202 8473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.397459 8458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:10.397583 8458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "8a5893db6eb540d1b7075b4038b32569"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "f4d4e2b858854bebf137f739d584653b"
server_key_iv: "767c8f407a84dd95ec4b087ca8fa1928"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.397697 8458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:10.416761 8458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:10.420007 8458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:10.420221 8458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:10.420831 8458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:10.421674 8458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:10.421746 8458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.421820 8458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:10.421864 8458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.431365 8458 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:37879
I20260619 22:26:10.431391 8586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:37879 every 8 connection(s)
I20260619 22:26:10.432288 8458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:10.442106 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8458
I20260619 22:26:10.442248 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:10.442543 5827 external_mini_cluster.cc:1428] Setting key defec89272af61c1db1ddd13ffae4f11
I20260619 22:26:10.446869 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:10.434287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:51377 (local address 127.5.176.254:43273)
0619 22:26:10.434457 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:10.434461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:10.435346 (+ 885us) server_negotiation.cc:407] Connection header received
0619 22:26:10.436801 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:10.436805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:10.436871 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:10.436995 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:10.438260 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.438820 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.440030 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.440278 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.440403 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:10.443082 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:10.443086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:10.443088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:10.443140 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:10.444648 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:10.445233 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:10.445238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:10.445244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:10.445321 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:10.445674 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:10.445677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:10.445678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:10.445872 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:10.445979 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:10.446576 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 22:26:10.446711 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:26:10.447748 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:10.434660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43273 (local address 127.5.176.194:51377)
0619 22:26:10.435152 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:10.435230 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:10.436555 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:10.437177 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:10.437186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:10.437568 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:10.438064 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:10.438077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.438959 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.438965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:10.439876 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:10.439887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.440444 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.440449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:10.440632 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:10.441253 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:10.441272 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:10.442889 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:10.444820 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:10.444827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:10.444843 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:10.445112 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:10.445423 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:10.445426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:10.445428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:10.445571 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:10.445990 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:10.445998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:10.446335 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 22:26:10.446523 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 22:26:10.446879 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260619 22:26:10.449014 8587 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43273
I20260619 22:26:10.449256 8587 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:10.449770 8587 heartbeater.cc:511] Master 127.5.176.254:43273 requested a full tablet report, sending...
I20260619 22:26:10.450862 8262 ts_manager.cc:194] Registered new tserver with Master: 8a5893db6eb540d1b7075b4038b32569 (127.5.176.194:37879)
I20260619 22:26:10.451457 8262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:51377
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:10.468900 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43273
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:10.579180 8592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:10.579457 8592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:10.579538 8592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:10.583549 8592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:10.583642 8592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:10.583729 8592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:10.589002 8592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:10.590080 8592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:10.591113 8592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:10.600692 8598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:10.600721 8597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:10.600828 8600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:10.600860 8592 server_base.cc:1034] running on GCE node
I20260619 22:26:10.601361 8592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:10.602059 8592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:10.603255 8592 hybrid_clock.cc:648] HybridClock initialized: now 1781907970603236 us; error 29 us; skew 500 ppm
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:10.605885 8592 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:10.606802 8592 webserver.cc:460] Webserver started at http://127.5.176.195:39591/ using document root <none> and password file <none>
I20260619 22:26:10.607350 8592 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:10.607447 8592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:10.607671 8592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:10.609712 8592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "29eb17d352d947beb2d535aac93b70ea"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "295be3578fbd46d2753e56e25432cfc7"
server_key_iv: "9c355e74b4c2b839e4f9918b8ae8e18f"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.610351 8592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "29eb17d352d947beb2d535aac93b70ea"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "295be3578fbd46d2753e56e25432cfc7"
server_key_iv: "9c355e74b4c2b839e4f9918b8ae8e18f"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.614099 8592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:26:10.616586 8607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.617772 8592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:10.617904 8592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "29eb17d352d947beb2d535aac93b70ea"
format_stamp: "Formatted at 2026-06-19 22:26:10 on dist-test-slave-bt27"
server_key: "295be3578fbd46d2753e56e25432cfc7"
server_key_iv: "9c355e74b4c2b839e4f9918b8ae8e18f"
server_key_version: "encryptionkey@0"
I20260619 22:26:10.618013 8592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:10.633673 8592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:10.636735 8592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:10.636945 8592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:10.637531 8592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:10.638396 8592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:10.638449 8592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.638489 8592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:10.638504 8592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:10.647377 8592 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:38049
I20260619 22:26:10.647398 8720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:38049 every 8 connection(s)
I20260619 22:26:10.648249 8592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:10.655642 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8592
I20260619 22:26:10.655810 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:10.656111 5827 external_mini_cluster.cc:1428] Setting key 0371c97da5976cf85f147cc87e18e5ed
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:10.661792 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:10.650029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:49943 (local address 127.5.176.254:43273)
0619 22:26:10.650202 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:10.650209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:10.651026 (+ 817us) server_negotiation.cc:407] Connection header received
0619 22:26:10.652358 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:10.652362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:10.652422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:10.652523 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:10.653616 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.654234 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.655401 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.655612 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.655724 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:10.658648 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:10.658651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:10.658653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:10.658675 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:10.659986 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:10.660477 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:10.660479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:10.660480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:10.660555 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:10.660880 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:10.660882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:10.660883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:10.661045 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:10.661118 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:10.661580 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 22:26:10.661678 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:10.662673 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:10.650463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43273 (local address 127.5.176.195:49943)
0619 22:26:10.650885 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:10.650925 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:10.652148 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:10.652684 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:10.652690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:10.653027 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:10.653477 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:10.653490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.654357 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.654365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:10.655220 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:10.655228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:10.655760 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:10.655765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:10.655980 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:10.656586 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:10.656609 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:10.658502 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:10.660097 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:10.660101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:10.660113 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:10.660385 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:10.660654 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:10.660657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:10.660659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:10.660797 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:10.661137 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:10.661141 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:10.661371 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 22:26:10.661536 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 22:26:10.661784 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260619 22:26:10.663753 8721 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43273
I20260619 22:26:10.663951 8721 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:10.664366 8721 heartbeater.cc:511] Master 127.5.176.254:43273 requested a full tablet report, sending...
I20260619 22:26:10.665294 8262 ts_manager.cc:194] Registered new tserver with Master: 29eb17d352d947beb2d535aac93b70ea (127.5.176.195:38049)
I20260619 22:26:10.665961 8262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:49943
I20260619 22:26:10.669808 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 22:26:10.682551 8301 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:10.681000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.5.176.254:43273)
0619 22:26:10.681148 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:10.681153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:10.681340 (+ 187us) server_negotiation.cc:407] Connection header received
0619 22:26:10.681415 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:10.681417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:10.681466 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:10.681541 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:10.682400 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33486: BlockingRecv error: recv got EOF from 127.0.0.1:33486 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:26:10.684216 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8324
I20260619 22:26:10.689602 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8458
I20260619 22:26:10.694630 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8592
I20260619 22:26:10.699894 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8230
2026-06-19T22:26:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2168 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8733](info): set up 2 sockets
Jun 19 22:26:10 dist-test-slave-bt27 krb5kdc[8733](info): commencing operation
krb5kdc: starting...
W20260619 22:26:11.738250 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:11 dist-test-slave-bt27 krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:11.858850 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38827
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:11.968189 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:11.968472 8746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:11.968514 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:11.972491 8746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:11.972556 8746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:11.972576 8746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:11.972594 8746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:11.972608 8746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:11.977802 8746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:38827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:11.978837 8746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:11.979820 8746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:11.988241 8754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:11.988327 8751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:11.988241 8752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:11.988510 8746 server_base.cc:1034] running on GCE node
I20260619 22:26:11.989022 8746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:11.990128 8746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:11.991353 8746 hybrid_clock.cc:648] HybridClock initialized: now 1781907971991339 us; error 24 us; skew 500 ppm
Jun 19 22:26:11 dist-test-slave-bt27 krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907971, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:11.994655 8746 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:11.995579 8746 webserver.cc:460] Webserver started at http://127.5.176.254:43763/ using document root <none> and password file <none>
I20260619 22:26:11.996122 8746 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:11.996182 8746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:11.996423 8746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:11.998605 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "e9bcbf8136d04047839b189b8aec9652"
format_stamp: "Formatted at 2026-06-19 22:26:11 on dist-test-slave-bt27"
server_key: "d7a9291c8651d0de540ea38f549ee6a5"
server_key_iv: "f907ac83f443241a421e4fc8eff1c9e8"
server_key_version: "encryptionkey@0"
I20260619 22:26:11.999178 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "e9bcbf8136d04047839b189b8aec9652"
format_stamp: "Formatted at 2026-06-19 22:26:11 on dist-test-slave-bt27"
server_key: "d7a9291c8651d0de540ea38f549ee6a5"
server_key_iv: "f907ac83f443241a421e4fc8eff1c9e8"
server_key_version: "encryptionkey@0"
I20260619 22:26:12.003255 8746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 22:26:12.005681 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:12.007139 8746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:12.007285 8746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "e9bcbf8136d04047839b189b8aec9652"
format_stamp: "Formatted at 2026-06-19 22:26:11 on dist-test-slave-bt27"
server_key: "d7a9291c8651d0de540ea38f549ee6a5"
server_key_iv: "f907ac83f443241a421e4fc8eff1c9e8"
server_key_version: "encryptionkey@0"
I20260619 22:26:12.007400 8746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:12.028494 8746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:12.031935 8746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:12.032153 8746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:12.039327 8746 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:38827
I20260619 22:26:12.039660 8813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:38827 every 8 connection(s)
I20260619 22:26:12.040337 8746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:12.043185 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:12.045032 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8746
I20260619 22:26:12.045163 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:12.045570 5827 external_mini_cluster.cc:1428] Setting key fd830336ac7bfaf47e2489a57eb4cc8f
I20260619 22:26:12.050534 8814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652: Bootstrap starting.
I20260619 22:26:12.053223 8814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:12.054214 8814 log.cc:824] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:26:12 dist-test-slave-bt27 krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:12.056756 8814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652: No bootstrap required, opened a new log
I20260619 22:26:12.061321 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:12.047249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.5.176.254:38827)
0619 22:26:12.047731 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:12.047749 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:12.047790 (+ 41us) server_negotiation.cc:407] Connection header received
0619 22:26:12.048481 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:12.048498 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:12.048854 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:12.049187 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:12.050233 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:12.051300 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:12.052514 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:12.053023 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:12.053152 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:12.055984 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:12.055990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:12.056008 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:12.056040 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:12.058032 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:12.058546 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:12.058552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:12.058559 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:12.058704 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:12.058959 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:12.058965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:12.058970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:12.059585 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:12.059768 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:12.060008 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 22:26:12.060311 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":160,"threads_started":1}
I20260619 22:26:12.061638 8814 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bcbf8136d04047839b189b8aec9652" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38827 } }
I20260619 22:26:12.061913 8814 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:12.061969 8814 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bcbf8136d04047839b189b8aec9652, State: Initialized, Role: FOLLOWER
I20260619 22:26:12.062489 8814 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bcbf8136d04047839b189b8aec9652" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38827 } }
I20260619 22:26:12.062614 8814 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:12.062696 8814 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:12.062798 8814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:12.063966 8814 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bcbf8136d04047839b189b8aec9652" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38827 } }
I20260619 22:26:12.064356 8814 leader_election.cc:302] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bcbf8136d04047839b189b8aec9652; no voters:
I20260619 22:26:12.064436 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:26:12.064720 8814 leader_election.cc:288] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:12.064841 8819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:12.065065 8819 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [term 1 LEADER]: Becoming Leader. State: Replica: e9bcbf8136d04047839b189b8aec9652, State: Running, Role: LEADER
I20260619 22:26:12.065461 8819 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bcbf8136d04047839b189b8aec9652" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38827 } }
I20260619 22:26:12.066275 8814 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:12.067422 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bcbf8136d04047839b189b8aec9652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bcbf8136d04047839b189b8aec9652" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38827 } } }
I20260619 22:26:12.067548 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:12.067979 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bcbf8136d04047839b189b8aec9652. Latest consensus state: current_term: 1 leader_uuid: "e9bcbf8136d04047839b189b8aec9652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bcbf8136d04047839b189b8aec9652" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38827 } } }
I20260619 22:26:12.068143 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:12.068485 8829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:12.071648 8829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:12.078096 8829 catalog_manager.cc:1347] Generated new cluster ID: 897ff4cc498d4cb58a2b19457eca8aea
I20260619 22:26:12.078238 8829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:12.084268 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:12.078503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.5.176.254:38827)
0619 22:26:12.078612 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:12.078616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:12.078917 (+ 301us) server_negotiation.cc:407] Connection header received
0619 22:26:12.079052 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:12.079055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:12.079112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:12.079209 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:12.079807 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:12.080635 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:12.081777 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:12.082202 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:12.082311 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:12.083209 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:12.083213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:12.083216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:12.083797 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:12.083832 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:12.083852 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:12.083946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:12.084046 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 22:26:12.084111 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 22:26:12.093443 8829 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:12.095407 8829 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:12.105237 8829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bcbf8136d04047839b189b8aec9652: Generated new TSK 0
I20260619 22:26:12.106259 8829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:12.116323 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8746
2026-06-19T22:26:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1414 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:12 dist-test-slave-bt27 krb5kdc[8847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:12 dist-test-slave-bt27 krb5kdc[8847](info): set up 2 sockets
Jun 19 22:26:12 dist-test-slave-bt27 krb5kdc[8847](info): commencing operation
krb5kdc: starting...
W20260619 22:26:13.162232 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:13.283366 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37559
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:13.394528 8860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:13.394802 8860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:13.394845 8860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:13.399008 8860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:13.399062 8860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:13.399081 8860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:13.399098 8860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:13.399163 8860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:13.404368 8860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:37559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:13.405586 8860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:13.406618 8860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:13.414841 8868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:13.414860 8865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:13.414914 8866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:13.415097 8860 server_base.cc:1034] running on GCE node
I20260619 22:26:13.415740 8860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:13.416870 8860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:13.418033 8860 hybrid_clock.cc:648] HybridClock initialized: now 1781907973418016 us; error 34 us; skew 500 ppm
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:13.420755 8860 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:13.421602 8860 webserver.cc:460] Webserver started at http://127.5.176.254:42061/ using document root <none> and password file <none>
I20260619 22:26:13.422122 8860 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:13.422262 8860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:13.422442 8860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:13.424471 8860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "05e20e698c874d3f92876a101419c28b"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "ed095582097e75b707e0c11e1603b9e8"
server_key_iv: "31a366705fc4e46e02c89374ea2fa268"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.424999 8860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "05e20e698c874d3f92876a101419c28b"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "ed095582097e75b707e0c11e1603b9e8"
server_key_iv: "31a366705fc4e46e02c89374ea2fa268"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.428746 8860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 22:26:13.431207 8875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.432418 8860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:13.432554 8860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "05e20e698c874d3f92876a101419c28b"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "ed095582097e75b707e0c11e1603b9e8"
server_key_iv: "31a366705fc4e46e02c89374ea2fa268"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.432662 8860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:13.450248 8860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:13.453087 8860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:13.453299 8860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:13.460397 8860 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:37559
I20260619 22:26:13.460719 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:37559 every 8 connection(s)
I20260619 22:26:13.461362 8860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:13.464116 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:13.469635 8928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b: Bootstrap starting.
I20260619 22:26:13.469738 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8860
I20260619 22:26:13.469852 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:13.470130 5827 external_mini_cluster.cc:1428] Setting key c7237fa823545f9d2dcaeb343c2993c2
I20260619 22:26:13.472484 8928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:13.473241 8928 log.cc:824] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:13.475257 8928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b: No bootstrap required, opened a new log
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:13.479853 8928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e20e698c874d3f92876a101419c28b" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37559 } }
I20260619 22:26:13.480129 8928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:13.480219 8928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e20e698c874d3f92876a101419c28b, State: Initialized, Role: FOLLOWER
I20260619 22:26:13.480738 8928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e20e698c874d3f92876a101419c28b" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37559 } }
I20260619 22:26:13.480929 8928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:13.481019 8928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:13.481149 8928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:13.482091 8928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e20e698c874d3f92876a101419c28b" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37559 } }
I20260619 22:26:13.482479 8928 leader_election.cc:302] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05e20e698c874d3f92876a101419c28b; no voters:
I20260619 22:26:13.482856 8928 leader_election.cc:288] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:13.482942 8933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:13.483126 8933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [term 1 LEADER]: Becoming Leader. State: Replica: 05e20e698c874d3f92876a101419c28b, State: Running, Role: LEADER
I20260619 22:26:13.483517 8933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e20e698c874d3f92876a101419c28b" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37559 } }
I20260619 22:26:13.484220 8928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:13.485491 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:13.471516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.5.176.254:37559)
0619 22:26:13.471986 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:13.472002 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:13.472037 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:26:13.472755 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:13.472774 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:13.473113 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:13.473465 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:13.474330 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.475312 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.476514 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.476975 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.477091 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:13.480016 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:13.480021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:13.480035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:13.480063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:13.481944 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:13.482494 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:13.482507 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:13.482512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:13.482655 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:13.483019 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:13.483023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:13.483028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:13.483580 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:13.483766 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:13.484165 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 22:26:13.484531 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260619 22:26:13.486327 8935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e20e698c874d3f92876a101419c28b. Latest consensus state: current_term: 1 leader_uuid: "05e20e698c874d3f92876a101419c28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e20e698c874d3f92876a101419c28b" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37559 } } }
I20260619 22:26:13.486737 8935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:13.486632 8934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05e20e698c874d3f92876a101419c28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e20e698c874d3f92876a101419c28b" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37559 } } }
I20260619 22:26:13.487246 8934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b [sys.catalog]: This master's current role is: LEADER
W20260619 22:26:13.489444 8948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:26:13.489530 8948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:26:13.489603 8949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:13.490393 8949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:13.495473 8949 catalog_manager.cc:1347] Generated new cluster ID: 6b72d0b64fb9488b936ebc4c2fee4f7f
I20260619 22:26:13.495546 8949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:13.523092 8949 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:13.524451 8949 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:13.529783 8949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05e20e698c874d3f92876a101419c28b: Generated new TSK 0
I20260619 22:26:13.530634 8949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:13.564778 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37559
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:13.680748 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:13.681051 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:13.681119 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:13.685371 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:13.685458 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:13.685541 8954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:13.690922 8954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:13.691998 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:13.692991 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:13.702461 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:13.702454 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:13.702503 8954 server_base.cc:1034] running on GCE node
W20260619 22:26:13.702454 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:13.703112 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:13.703743 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:13.704921 8954 hybrid_clock.cc:648] HybridClock initialized: now 1781907973704884 us; error 46 us; skew 500 ppm
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:13.711078 8954 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:13.712069 8954 webserver.cc:460] Webserver started at http://127.5.176.193:41651/ using document root <none> and password file <none>
I20260619 22:26:13.712641 8954 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:13.712740 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:13.712972 8954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:13.715085 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "767136d84bac47268f372d3be055302a"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "3417ec5f5b7ed89b3f180cd3f945ac45"
server_key_iv: "53b20fa52842446e995a6f8f8a7f4859"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.715644 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "767136d84bac47268f372d3be055302a"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "3417ec5f5b7ed89b3f180cd3f945ac45"
server_key_iv: "53b20fa52842446e995a6f8f8a7f4859"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.719502 8954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 22:26:13.721882 8969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.723207 8954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 22:26:13.723356 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "767136d84bac47268f372d3be055302a"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "3417ec5f5b7ed89b3f180cd3f945ac45"
server_key_iv: "53b20fa52842446e995a6f8f8a7f4859"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.723479 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:13.735940 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:13.739545 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:13.739759 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:13.740363 8954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:13.741204 8954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:13.741273 8954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.741346 8954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:13.741389 8954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.750108 8954 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:38591
I20260619 22:26:13.750176 9082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:38591 every 8 connection(s)
I20260619 22:26:13.751056 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:13.751691 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 8954
I20260619 22:26:13.751822 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:13.752071 5827 external_mini_cluster.cc:1428] Setting key 1e3dc6757154f2b1153226f9d36f866f
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:13.765163 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:13.752944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:33557 (local address 127.5.176.254:37559)
0619 22:26:13.753086 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:13.753091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:13.753923 (+ 832us) server_negotiation.cc:407] Connection header received
0619 22:26:13.755334 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:13.755337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:13.755391 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:13.755480 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:13.756637 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.757187 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.758451 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.758695 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.758814 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:13.761369 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:13.761372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:13.761374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:13.761400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:13.762890 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:13.763523 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:13.763527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:13.763529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:13.763608 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:13.763972 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:13.763977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:13.763983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:13.764160 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:13.764268 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:13.764924 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 22:26:13.765036 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:26:13.766340 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:13.753341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37559 (local address 127.5.176.193:33557)
0619 22:26:13.753789 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:13.753831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:13.755134 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:13.755651 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:13.755658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:13.756019 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:13.756488 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:13.756501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.757319 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.757324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:13.758319 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:13.758329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.758855 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.758859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:13.759052 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:13.759696 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:13.759713 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:13.761174 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:13.763020 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:13.763026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:13.763060 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:13.763389 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:13.763711 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:13.763713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:13.763715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:13.763856 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:13.764265 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:13.764272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:13.764636 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 22:26:13.764873 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 22:26:13.765277 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260619 22:26:13.767630 9083 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37559
I20260619 22:26:13.767889 9083 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:13.768399 9083 heartbeater.cc:511] Master 127.5.176.254:37559 requested a full tablet report, sending...
I20260619 22:26:13.769968 8889 ts_manager.cc:194] Registered new tserver with Master: 767136d84bac47268f372d3be055302a (127.5.176.193:38591)
I20260619 22:26:13.771153 8889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:33557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:13.779106 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37559
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:13.892135 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:13.892410 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:13.892455 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:13.896399 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:13.896450 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:13.896508 9088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:13.901660 9088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:13.902683 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:13.903640 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:13.913185 9096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:13.913192 9093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:13.913192 9094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:13.913465 9088 server_base.cc:1034] running on GCE node
I20260619 22:26:13.913944 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:13.914690 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:13.915881 9088 hybrid_clock.cc:648] HybridClock initialized: now 1781907973915871 us; error 36 us; skew 500 ppm
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:13.918732 9088 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:13.919633 9088 webserver.cc:460] Webserver started at http://127.5.176.194:35663/ using document root <none> and password file <none>
I20260619 22:26:13.920229 9088 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:13.920323 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:13.920559 9088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:13.922690 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "fae8bf3fc3ae435fa84f3c9d1a72542f"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "ace003ba4f7d51a0a79afdfcbdeb7c8c"
server_key_iv: "f83c84e8284befedec584d0ed68a3a02"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.923266 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "fae8bf3fc3ae435fa84f3c9d1a72542f"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "ace003ba4f7d51a0a79afdfcbdeb7c8c"
server_key_iv: "f83c84e8284befedec584d0ed68a3a02"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.927232 9088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:13.929745 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.931087 9088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:13.931224 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "fae8bf3fc3ae435fa84f3c9d1a72542f"
format_stamp: "Formatted at 2026-06-19 22:26:13 on dist-test-slave-bt27"
server_key: "ace003ba4f7d51a0a79afdfcbdeb7c8c"
server_key_iv: "f83c84e8284befedec584d0ed68a3a02"
server_key_version: "encryptionkey@0"
I20260619 22:26:13.931339 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:13.949574 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:13.953027 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:13.953245 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:13.953926 9088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:13.954813 9088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:13.954886 9088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.954962 9088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:13.955004 9088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:13.964021 9088 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:44137
I20260619 22:26:13.964042 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:44137 every 8 connection(s)
I20260619 22:26:13.964943 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:13.965694 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9088
I20260619 22:26:13.965826 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:13.966121 5827 external_mini_cluster.cc:1428] Setting key 86ca299065577b8a8db0d7d697c156a6
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
Jun 19 22:26:13 dist-test-slave-bt27 krb5kdc[8847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:26:13.979417 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:13.966875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:58877 (local address 127.5.176.254:37559)
0619 22:26:13.967016 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:13.967019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:13.968055 (+ 1036us) server_negotiation.cc:407] Connection header received
0619 22:26:13.969436 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:13.969441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:13.969501 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:13.969600 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:13.970891 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.971427 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.972568 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.972798 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.972907 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:13.975613 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:13.975617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:13.975619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:13.975643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:13.977144 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:13.977753 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:13.977758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:13.977763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:13.977859 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:13.978218 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:13.978224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:13.978229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:13.978495 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:13.978604 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:13.979198 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 22:26:13.979297 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:26:13.980600 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:13.967390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37559 (local address 127.5.176.194:58877)
0619 22:26:13.967882 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:13.967944 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:13.969235 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:13.969785 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:13.969792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:13.970209 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:13.970717 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:13.970730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.971551 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.971555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:13.972440 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:13.972449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:13.972942 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:13.972945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:13.973124 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:13.973711 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:13.973727 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:13.975426 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:13.977274 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:13.977282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:13.977300 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:13.977657 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:13.977959 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:13.977961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:13.977962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:13.978085 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:13.978605 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:13.978610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:13.978927 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 22:26:13.979134 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 22:26:13.979487 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260619 22:26:13.981863 9217 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37559
I20260619 22:26:13.982113 9217 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:13.982630 9217 heartbeater.cc:511] Master 127.5.176.254:37559 requested a full tablet report, sending...
I20260619 22:26:13.983704 8889 ts_manager.cc:194] Registered new tserver with Master: fae8bf3fc3ae435fa84f3c9d1a72542f (127.5.176.194:44137)
I20260619 22:26:13.984272 8889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:58877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:13.993233 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37559
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:14.110342 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:14.110648 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:14.110759 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:14.114912 9222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:14.115005 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:14.115097 9222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:14.121191 9222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:14.122443 9222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:14.123503 9222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:14.133838 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:14.134086 9222 server_base.cc:1034] running on GCE node
W20260619 22:26:14.133841 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:14.133838 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:14.134742 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:14.135530 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:14.136694 9222 hybrid_clock.cc:648] HybridClock initialized: now 1781907974136647 us; error 103 us; skew 500 ppm
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[8847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907974, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:14.139770 9222 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:14.140712 9222 webserver.cc:460] Webserver started at http://127.5.176.195:43785/ using document root <none> and password file <none>
I20260619 22:26:14.141312 9222 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:14.141407 9222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:14.141657 9222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:14.143862 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "997bfbd34bc04bd4bc44c14570160ce3"
format_stamp: "Formatted at 2026-06-19 22:26:14 on dist-test-slave-bt27"
server_key: "d03ace3e3b73944db87a75a4806f04e2"
server_key_iv: "2d810e81ac044da7bcbcc2c1a723c51d"
server_key_version: "encryptionkey@0"
I20260619 22:26:14.144481 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "997bfbd34bc04bd4bc44c14570160ce3"
format_stamp: "Formatted at 2026-06-19 22:26:14 on dist-test-slave-bt27"
server_key: "d03ace3e3b73944db87a75a4806f04e2"
server_key_iv: "2d810e81ac044da7bcbcc2c1a723c51d"
server_key_version: "encryptionkey@0"
I20260619 22:26:14.148501 9222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 22:26:14.151119 9237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:14.152415 9222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:14.152568 9222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "997bfbd34bc04bd4bc44c14570160ce3"
format_stamp: "Formatted at 2026-06-19 22:26:14 on dist-test-slave-bt27"
server_key: "d03ace3e3b73944db87a75a4806f04e2"
server_key_iv: "2d810e81ac044da7bcbcc2c1a723c51d"
server_key_version: "encryptionkey@0"
I20260619 22:26:14.152694 9222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:14.166060 9222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:14.169198 9222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:14.169425 9222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:14.170049 9222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:14.170954 9222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:14.171046 9222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:14.171119 9222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:14.171164 9222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:14.179888 9222 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:35931
I20260619 22:26:14.179967 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:35931 every 8 connection(s)
I20260619 22:26:14.180859 9222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:14.190014 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9222
I20260619 22:26:14.190218 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:14.190562 5827 external_mini_cluster.cc:1428] Setting key fa10e4141159be6792505f8eaa452ec8
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[8847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907974, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:14.196600 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:14.182791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:39371 (local address 127.5.176.254:37559)
0619 22:26:14.182968 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:14.182973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:14.183798 (+ 825us) server_negotiation.cc:407] Connection header received
0619 22:26:14.185309 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:14.185312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:14.185363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:14.185441 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:14.186817 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:14.187352 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:14.189070 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:14.189286 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:14.189373 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:14.192311 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:14.192315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:14.192318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:14.192348 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:14.194294 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:14.194889 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:14.194895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:14.194897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:14.194998 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:14.195390 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:14.195394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:14.195396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:14.195613 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:14.195744 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:14.196282 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 22:26:14.196417 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:26:14.197456 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:14.183172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37559 (local address 127.5.176.195:39371)
0619 22:26:14.183634 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:14.183678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:14.185121 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:14.185732 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:14.185740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:14.186119 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:14.186666 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:14.186680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:14.187475 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:14.187478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:14.188919 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:14.188948 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:14.189427 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:14.189431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:14.189572 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:14.190280 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:14.190303 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:14.192144 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:14.194429 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:14.194437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:14.194455 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:14.194770 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:14.195104 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:14.195107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:14.195110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:14.195277 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:14.195743 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:14.195751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:14.196023 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 22:26:14.196242 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 22:26:14.196601 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260619 22:26:14.198617 9351 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37559
I20260619 22:26:14.198904 9351 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:14.199402 9351 heartbeater.cc:511] Master 127.5.176.254:37559 requested a full tablet report, sending...
I20260619 22:26:14.200434 8889 ts_manager.cc:194] Registered new tserver with Master: 997bfbd34bc04bd4bc44c14570160ce3 (127.5.176.195:35931)
I20260619 22:26:14.201025 8889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:39371
I20260619 22:26:14.204548 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[8847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907974, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[8847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907974, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:14.232009 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:14.221960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.5.176.254:37559)
0619 22:26:14.222091 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:14.222095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:14.222306 (+ 211us) server_negotiation.cc:407] Connection header received
0619 22:26:14.222496 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:14.222503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:14.222560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:14.222669 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:14.223604 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:14.224187 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:14.225484 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:14.225800 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:14.225923 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:14.228743 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:14.228746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:14.228749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:14.228779 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:14.230328 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:14.230762 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:14.230765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:14.230766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:14.230868 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:14.231104 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:14.231106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:14.231107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:14.231359 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:14.231467 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:14.231661 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 22:26:14.231764 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 22:26:14.232429 8889 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48606
I20260619 22:26:14.233484 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8954
I20260619 22:26:14.239660 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9088
I20260619 22:26:14.245553 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9222
I20260619 22:26:14.250844 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 8860
2026-06-19T22:26:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2135 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[9363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[9363](info): set up 2 sockets
Jun 19 22:26:14 dist-test-slave-bt27 krb5kdc[9363](info): commencing operation
krb5kdc: starting...
W20260619 22:26:15.302245 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:15 dist-test-slave-bt27 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:15.425030 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:41921
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:15.537529 9376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:15.537817 9376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:15.537859 9376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:15.541989 9376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:15.542047 9376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:15.542068 9376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:15.542084 9376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:15.542099 9376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:15.547649 9376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:41921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:15.548729 9376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:15.549764 9376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:15.558354 9384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:15.558437 9376 server_base.cc:1034] running on GCE node
W20260619 22:26:15.558382 9382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:15.558354 9381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:15.559134 9376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:15.560277 9376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:15.561444 9376 hybrid_clock.cc:648] HybridClock initialized: now 1781907975561413 us; error 46 us; skew 500 ppm
Jun 19 22:26:15 dist-test-slave-bt27 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:15.564325 9376 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:15.565250 9376 webserver.cc:460] Webserver started at http://127.5.176.254:38851/ using document root <none> and password file <none>
I20260619 22:26:15.565807 9376 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:15.565917 9376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:15.566195 9376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:15.568248 9376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "ffcf96909b6142339c97ac2c302ac9d8"
format_stamp: "Formatted at 2026-06-19 22:26:15 on dist-test-slave-bt27"
server_key: "44e348e1bc80c0e0001fde66202ded03"
server_key_iv: "a01867ff82082bf4dfb9705c3ab53f6e"
server_key_version: "encryptionkey@0"
I20260619 22:26:15.568821 9376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "ffcf96909b6142339c97ac2c302ac9d8"
format_stamp: "Formatted at 2026-06-19 22:26:15 on dist-test-slave-bt27"
server_key: "44e348e1bc80c0e0001fde66202ded03"
server_key_iv: "a01867ff82082bf4dfb9705c3ab53f6e"
server_key_version: "encryptionkey@0"
I20260619 22:26:15.572655 9376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:26:15.575254 9391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:15.576747 9376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:15.576906 9376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "ffcf96909b6142339c97ac2c302ac9d8"
format_stamp: "Formatted at 2026-06-19 22:26:15 on dist-test-slave-bt27"
server_key: "44e348e1bc80c0e0001fde66202ded03"
server_key_iv: "a01867ff82082bf4dfb9705c3ab53f6e"
server_key_version: "encryptionkey@0"
I20260619 22:26:15.577030 9376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:15.607197 9376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:15.610554 9376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:15.610752 9376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:15.618139 9376 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:41921
I20260619 22:26:15.618181 9443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:41921 every 8 connection(s)
I20260619 22:26:15.619160 9376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:15.621769 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9376
I20260619 22:26:15.621882 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:15.621996 9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:15.622268 5827 external_mini_cluster.cc:1428] Setting key 6ec962cb96aaeaca2a35f44c0a07c729
I20260619 22:26:15.629480 9444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8: Bootstrap starting.
Jun 19 22:26:15 dist-test-slave-bt27 krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:15.632251 9444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:15.633105 9444 log.cc:824] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:15.635547 9444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8: No bootstrap required, opened a new log
I20260619 22:26:15.638175 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:15.623696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.5.176.254:41921)
0619 22:26:15.624150 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:15.624168 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:15.624210 (+ 42us) server_negotiation.cc:407] Connection header received
0619 22:26:15.624896 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:15.624919 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:15.625348 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:15.625741 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:15.626702 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:15.628224 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:15.629116 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:15.629798 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:15.629945 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:15.632797 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:15.632804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:15.632821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:15.632854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:15.634948 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:15.635413 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:15.635418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:15.635424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:15.635554 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:15.635896 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:15.635901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:15.635904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:15.636391 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:15.636581 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:15.636807 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 22:26:15.637094 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":132,"threads_started":1}
I20260619 22:26:15.640692 9444 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcf96909b6142339c97ac2c302ac9d8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41921 } }
I20260619 22:26:15.641047 9444 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:15.641134 9444 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcf96909b6142339c97ac2c302ac9d8, State: Initialized, Role: FOLLOWER
I20260619 22:26:15.641633 9444 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcf96909b6142339c97ac2c302ac9d8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41921 } }
I20260619 22:26:15.641794 9444 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:15.641855 9444 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:15.641952 9444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:15.643229 9444 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcf96909b6142339c97ac2c302ac9d8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41921 } }
I20260619 22:26:15.643594 9444 leader_election.cc:302] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffcf96909b6142339c97ac2c302ac9d8; no voters:
I20260619 22:26:15.643944 9444 leader_election.cc:288] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:15.644099 9449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:15.644284 9449 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [term 1 LEADER]: Becoming Leader. State: Replica: ffcf96909b6142339c97ac2c302ac9d8, State: Running, Role: LEADER
I20260619 22:26:15.644586 9449 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcf96909b6142339c97ac2c302ac9d8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41921 } }
I20260619 22:26:15.645238 9444 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:15.645720 9450 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffcf96909b6142339c97ac2c302ac9d8. Latest consensus state: current_term: 1 leader_uuid: "ffcf96909b6142339c97ac2c302ac9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcf96909b6142339c97ac2c302ac9d8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41921 } } }
I20260619 22:26:15.645896 9450 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:15.646544 9456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:15.647559 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffcf96909b6142339c97ac2c302ac9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcf96909b6142339c97ac2c302ac9d8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41921 } } }
I20260619 22:26:15.647658 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:15.649876 9456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:15.655534 9456 catalog_manager.cc:1347] Generated new cluster ID: fbd403ee533a49878853b8db8fd5e974
I20260619 22:26:15.655603 9456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:15.674897 9456 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:15.675899 9456 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:15.685460 9456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffcf96909b6142339c97ac2c302ac9d8: Generated new TSK 0
I20260619 22:26:15.686314 9456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:15.717042 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41921
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:15.847779 9470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:15.848057 9470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:15.848098 9470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:15.852169 9470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:15.852226 9470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:15.852394 9470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:15.857699 9470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:15.858844 9470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:15.859899 9470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:15.869586 9476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:15.869626 9470 server_base.cc:1034] running on GCE node
W20260619 22:26:15.869594 9475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:15.869781 9478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:15.870368 9470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:15.870980 9470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:15.872156 9470 hybrid_clock.cc:648] HybridClock initialized: now 1781907975872129 us; error 47 us; skew 500 ppm
Jun 19 22:26:15 dist-test-slave-bt27 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:15.875125 9470 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:15.875974 9470 webserver.cc:460] Webserver started at http://127.5.176.193:35973/ using document root <none> and password file <none>
I20260619 22:26:15.876504 9470 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:15.876565 9470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:15.876740 9470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:15.878928 9470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "e95829921e514009abb590f0474b18c5"
format_stamp: "Formatted at 2026-06-19 22:26:15 on dist-test-slave-bt27"
server_key: "04fe1a6e7eefc22f83b7dbe9ae26a4be"
server_key_iv: "902123ec812f00467e87c3b837c54403"
server_key_version: "encryptionkey@0"
I20260619 22:26:15.879501 9470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "e95829921e514009abb590f0474b18c5"
format_stamp: "Formatted at 2026-06-19 22:26:15 on dist-test-slave-bt27"
server_key: "04fe1a6e7eefc22f83b7dbe9ae26a4be"
server_key_iv: "902123ec812f00467e87c3b837c54403"
server_key_version: "encryptionkey@0"
I20260619 22:26:15.883626 9470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:15.886071 9485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:15.887389 9470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 22:26:15.887485 9470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "e95829921e514009abb590f0474b18c5"
format_stamp: "Formatted at 2026-06-19 22:26:15 on dist-test-slave-bt27"
server_key: "04fe1a6e7eefc22f83b7dbe9ae26a4be"
server_key_iv: "902123ec812f00467e87c3b837c54403"
server_key_version: "encryptionkey@0"
I20260619 22:26:15.887573 9470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:15.906855 9470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:15.909684 9470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:15.909844 9470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:15.910472 9470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:15.911314 9470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:15.911358 9470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:15.911399 9470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:15.911418 9470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:15.920534 9470 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:33581
I20260619 22:26:15.920560 9598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:33581 every 8 connection(s)
I20260619 22:26:15.921487 9470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:15.924559 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9470
I20260619 22:26:15.924687 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:15.925056 5827 external_mini_cluster.cc:1428] Setting key 2ed4304454c5e805a99df1c3840c8e94
Jun 19 22:26:15 dist-test-slave-bt27 krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:26:15.935963 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:15.923375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:40421 (local address 127.5.176.254:41921)
0619 22:26:15.923557 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:15.923561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:15.924422 (+ 861us) server_negotiation.cc:407] Connection header received
0619 22:26:15.926243 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:15.926246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:15.926298 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:15.926381 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:15.927565 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:15.928093 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:15.929233 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:15.929468 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:15.929581 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:15.932087 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:15.932093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:15.932100 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:15.932124 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:15.933668 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:15.934365 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:15.934368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:15.934369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:15.934446 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:15.934885 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:15.934889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:15.934890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:15.935084 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:15.935161 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:15.935726 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 22:26:15.935837 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:15.937032 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:15.923745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41921 (local address 127.5.176.193:40421)
0619 22:26:15.924265 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:15.924317 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:15.925976 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:15.926579 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:15.926586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:15.926941 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:15.927422 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:15.927437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:15.928237 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:15.928240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:15.929109 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:15.929118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:15.929620 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:15.929624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:15.929794 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:15.930465 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:15.930482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:15.931916 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:15.933832 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:15.933839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:15.933856 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:15.934259 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:15.934535 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:15.934537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:15.934538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:15.934766 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:15.935175 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:15.935180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:15.935450 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 22:26:15.935664 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 22:26:15.935976 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":162,"threads_started":1}
I20260619 22:26:15.938429 9599 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41921
I20260619 22:26:15.938699 9599 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:15.939308 9599 heartbeater.cc:511] Master 127.5.176.254:41921 requested a full tablet report, sending...
I20260619 22:26:15.941140 9408 ts_manager.cc:194] Registered new tserver with Master: e95829921e514009abb590f0474b18c5 (127.5.176.193:33581)
I20260619 22:26:15.942864 9408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:40421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:15.953881 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41921
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:16.080049 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:16.080325 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:16.080370 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:16.084573 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:16.084647 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:16.084708 9604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:16.089949 9604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:16.091105 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:16.092219 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:16.101897 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:16.101938 9609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:16.101903 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:16.102105 9604 server_base.cc:1034] running on GCE node
I20260619 22:26:16.102717 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:16.103389 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:16.104604 9604 hybrid_clock.cc:648] HybridClock initialized: now 1781907976104568 us; error 51 us; skew 500 ppm
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:16.107967 9604 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:16.108990 9604 webserver.cc:460] Webserver started at http://127.5.176.194:46839/ using document root <none> and password file <none>
I20260619 22:26:16.109601 9604 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:16.109697 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:16.109938 9604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:16.112272 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "7e67e840030a46c4964f8ef0aae3bda9"
format_stamp: "Formatted at 2026-06-19 22:26:16 on dist-test-slave-bt27"
server_key: "3c66e5f4b583fbc0d828aeee8b33163f"
server_key_iv: "61f4470aed281f5a8c1b0dea12442941"
server_key_version: "encryptionkey@0"
I20260619 22:26:16.112993 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "7e67e840030a46c4964f8ef0aae3bda9"
format_stamp: "Formatted at 2026-06-19 22:26:16 on dist-test-slave-bt27"
server_key: "3c66e5f4b583fbc0d828aeee8b33163f"
server_key_iv: "61f4470aed281f5a8c1b0dea12442941"
server_key_version: "encryptionkey@0"
I20260619 22:26:16.116955 9604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:26:16.119638 9619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:16.120992 9604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:26:16.121138 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "7e67e840030a46c4964f8ef0aae3bda9"
format_stamp: "Formatted at 2026-06-19 22:26:16 on dist-test-slave-bt27"
server_key: "3c66e5f4b583fbc0d828aeee8b33163f"
server_key_iv: "61f4470aed281f5a8c1b0dea12442941"
server_key_version: "encryptionkey@0"
I20260619 22:26:16.121268 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:16.148154 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:16.151288 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:16.151492 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:16.152082 9604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:16.152885 9604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:16.152953 9604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:16.153039 9604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:16.153081 9604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:16.162024 9604 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:45777
I20260619 22:26:16.162042 9732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:45777 every 8 connection(s)
I20260619 22:26:16.162945 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:16.171650 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9604
I20260619 22:26:16.171756 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:16.172036 5827 external_mini_cluster.cc:1428] Setting key 164ccfde9fa9d1eaf20284c4a1193c15
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:16.177212 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:16.164766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:39155 (local address 127.5.176.254:41921)
0619 22:26:16.164914 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:16.164918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:16.165747 (+ 829us) server_negotiation.cc:407] Connection header received
0619 22:26:16.167247 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:16.167250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:16.167300 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:16.167384 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:16.168539 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.169130 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.170333 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.170560 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.170653 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:16.173360 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:16.173364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:16.173367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:16.173393 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:16.175165 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.175693 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.175696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.175698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.175782 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.176116 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.176120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.176122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.176317 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:16.176420 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:16.176936 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 22:26:16.177055 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:26:16.178053 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:16.165126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41921 (local address 127.5.176.194:39155)
0619 22:26:16.165584 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:16.165627 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:16.167010 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:16.167544 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:16.167551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:16.167908 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:16.168384 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:16.168397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.169244 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.169246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:16.170130 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:16.170204 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.170722 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.170726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:16.170899 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:16.171455 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:16.171471 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:16.173177 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:16.175288 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:16.175294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:16.175306 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:16.175573 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:16.175876 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:16.175879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:16.175881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:16.176018 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:16.176428 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:16.176437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:16.176680 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 22:26:16.176896 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 22:26:16.177217 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260619 22:26:16.179427 9733 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41921
I20260619 22:26:16.179673 9733 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:16.180178 9733 heartbeater.cc:511] Master 127.5.176.254:41921 requested a full tablet report, sending...
I20260619 22:26:16.181358 9408 ts_manager.cc:194] Registered new tserver with Master: 7e67e840030a46c4964f8ef0aae3bda9 (127.5.176.194:45777)
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:26:16.182231 9408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:39155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:16.201923 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41921
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:16.318810 9738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:16.319090 9738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:16.319136 9738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:16.323279 9738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:16.323352 9738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:16.323416 9738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:16.328918 9738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:16.329967 9738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:16.331017 9738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:16.340826 9744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:16.340801 9743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:16.341132 9738 server_base.cc:1034] running on GCE node
W20260619 22:26:16.340808 9746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:16.341616 9738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:16.342356 9738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:16.343555 9738 hybrid_clock.cc:648] HybridClock initialized: now 1781907976343523 us; error 52 us; skew 500 ppm
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:16.346853 9738 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:16.347795 9738 webserver.cc:460] Webserver started at http://127.5.176.195:38601/ using document root <none> and password file <none>
I20260619 22:26:16.348436 9738 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:16.348527 9738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:16.348762 9738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:16.350899 9738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "32405d3b4c884ef9a3bcad859620cabc"
format_stamp: "Formatted at 2026-06-19 22:26:16 on dist-test-slave-bt27"
server_key: "cd07a693acd1be4322944d80b0be4c0a"
server_key_iv: "cf1883429e1107a6c1e8db99ebbd74d2"
server_key_version: "encryptionkey@0"
I20260619 22:26:16.351477 9738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "32405d3b4c884ef9a3bcad859620cabc"
format_stamp: "Formatted at 2026-06-19 22:26:16 on dist-test-slave-bt27"
server_key: "cd07a693acd1be4322944d80b0be4c0a"
server_key_iv: "cf1883429e1107a6c1e8db99ebbd74d2"
server_key_version: "encryptionkey@0"
I20260619 22:26:16.355314 9738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:26:16.357792 9753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:16.359187 9738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:16.359337 9738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "32405d3b4c884ef9a3bcad859620cabc"
format_stamp: "Formatted at 2026-06-19 22:26:16 on dist-test-slave-bt27"
server_key: "cd07a693acd1be4322944d80b0be4c0a"
server_key_iv: "cf1883429e1107a6c1e8db99ebbd74d2"
server_key_version: "encryptionkey@0"
I20260619 22:26:16.359455 9738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:16.381872 9738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:16.385313 9738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:16.385545 9738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:16.386214 9738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:16.387084 9738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:16.387154 9738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:16.387225 9738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:16.387287 9738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:16.396445 9738 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:43005
I20260619 22:26:16.396461 9866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:43005 every 8 connection(s)
I20260619 22:26:16.397344 9738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:16.399231 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9738
I20260619 22:26:16.399364 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:16.399667 5827 external_mini_cluster.cc:1428] Setting key e72d8cb986fb946908be67aa9a946620
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:16.411710 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:16.399152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:46941 (local address 127.5.176.254:41921)
0619 22:26:16.399292 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:16.399299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:16.400338 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 22:26:16.402003 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:16.402007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:16.402062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:16.402176 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:16.403366 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.403903 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.405102 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.405334 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.405449 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:16.408189 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:16.408192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:16.408194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:16.408218 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:16.409556 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.410101 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.410105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.410106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.410239 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.410581 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.410586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.410588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.410792 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:16.410895 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:16.411417 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 22:26:16.411547 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:26:16.412660 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:16.399639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41921 (local address 127.5.176.195:46941)
0619 22:26:16.400186 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:16.400245 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:16.401808 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:16.402346 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:16.402353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:16.402713 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:16.403203 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:16.403218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.404053 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.404055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:16.404968 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:16.404977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.405495 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.405499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:16.405668 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:16.406376 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:16.406394 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:16.407996 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:16.409686 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:16.409692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:16.409709 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:16.409997 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:16.410338 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:16.410342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:16.410344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:16.410485 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:16.410905 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:16.410914 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:16.411176 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 22:26:16.411379 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 22:26:16.411720 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":148,"threads_started":1}
I20260619 22:26:16.413815 9867 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41921
I20260619 22:26:16.414036 9867 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:16.414546 9867 heartbeater.cc:511] Master 127.5.176.254:41921 requested a full tablet report, sending...
I20260619 22:26:16.415602 9407 ts_manager.cc:194] Registered new tserver with Master: 32405d3b4c884ef9a3bcad859620cabc (127.5.176.195:43005)
I20260619 22:26:16.416276 9407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:46941
I20260619 22:26:16.425830 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:26:16.452036 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:16.442266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.5.176.193:33581)
0619 22:26:16.442584 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:16.442593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:16.442626 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:26:16.442677 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:16.442682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:16.442889 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:16.443012 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:16.443743 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.444539 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.445506 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.445957 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.446069 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:16.448643 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:16.448652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:16.448665 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:16.448690 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:16.450515 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.450929 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.450934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.450935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.451009 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.451224 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.451226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.451227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.451461 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:16.451615 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:16.451764 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 22:26:16.451898 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":148,"threads_started":1}
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:16.463157 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:16.454901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.5.176.254:41921)
0619 22:26:16.455041 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:16.455045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:16.455058 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:16.455180 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:16.455183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:16.455222 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:16.455289 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:16.455847 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.456356 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.457086 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:16.457292 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:16.457367 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:16.459843 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:16.459847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:16.459849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:16.459883 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:16.461335 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.461787 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.461791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.461793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.461882 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:16.462194 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:16.462199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:16.462201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:16.462452 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:16.462600 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:16.462840 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 22:26:16.462986 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 22:26:16.463598 9407 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58920
I20260619 22:26:16.464288 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9470
I20260619 22:26:16.470414 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9604
I20260619 22:26:16.475715 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9738
I20260619 22:26:16.481206 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9376
2026-06-19T22:26:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2229 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9879](info): set up 2 sockets
Jun 19 22:26:16 dist-test-slave-bt27 krb5kdc[9879](info): commencing operation
krb5kdc: starting...
W20260619 22:26:17.526456 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:17 dist-test-slave-bt27 krb5kdc[9879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:17.649741 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:42985
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:17.760465 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:17.760754 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:17.760798 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:17.764986 9892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:17.765054 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:17.765074 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:17.765091 9892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:17.765106 9892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:17.770498 9892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:17.771605 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:17.772657 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:17.780956 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:17.780911 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:17.780964 9892 server_base.cc:1034] running on GCE node
W20260619 22:26:17.780911 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:17.781724 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:17.782861 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:17.784021 9892 hybrid_clock.cc:648] HybridClock initialized: now 1781907977783987 us; error 48 us; skew 500 ppm
Jun 19 22:26:17 dist-test-slave-bt27 krb5kdc[9879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907977, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:17.786791 9892 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:17.787495 9892 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:17.787556 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:17.787765 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:17.789824 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "3d342a06e5ff46308e5c85b07b043514"
format_stamp: "Formatted at 2026-06-19 22:26:17 on dist-test-slave-bt27"
server_key: "b42f8cd2592b447601522b254ac91633"
server_key_iv: "675173c46d5b25529f15ebc8773c7525"
server_key_version: "encryptionkey@0"
I20260619 22:26:17.790414 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "3d342a06e5ff46308e5c85b07b043514"
format_stamp: "Formatted at 2026-06-19 22:26:17 on dist-test-slave-bt27"
server_key: "b42f8cd2592b447601522b254ac91633"
server_key_iv: "675173c46d5b25529f15ebc8773c7525"
server_key_version: "encryptionkey@0"
I20260619 22:26:17.793994 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:26:17.796352 9906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:17.797537 9892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:17.797673 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "3d342a06e5ff46308e5c85b07b043514"
format_stamp: "Formatted at 2026-06-19 22:26:17 on dist-test-slave-bt27"
server_key: "b42f8cd2592b447601522b254ac91633"
server_key_iv: "675173c46d5b25529f15ebc8773c7525"
server_key_version: "encryptionkey@0"
I20260619 22:26:17.797782 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:17.811124 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:17.814236 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:17.814455 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:17.821851 9892 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:42985
I20260619 22:26:17.822129 9958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:42985 every 8 connection(s)
I20260619 22:26:17.822597 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:17.825793 9959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:17.826376 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9892
I20260619 22:26:17.826527 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:17.826789 5827 external_mini_cluster.cc:1428] Setting key 9e05a6f873016e5c2b78010f60e33c19
I20260619 22:26:17.834002 9959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514: Bootstrap starting.
Jun 19 22:26:17 dist-test-slave-bt27 krb5kdc[9879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:17.836902 9959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:17.837813 9959 log.cc:824] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:17.840003 9959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514: No bootstrap required, opened a new log
I20260619 22:26:17.841881 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:17.828110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.5.176.254:42985)
0619 22:26:17.828509 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:17.828523 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:17.828558 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:26:17.829398 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:17.829415 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:17.829742 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:17.830021 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:17.830995 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:17.832044 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:17.833277 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:17.833796 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:17.833930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:17.836800 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:17.836805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:17.836821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:17.836852 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:17.838921 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:17.839423 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:17.839429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:17.839433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:17.839564 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:17.839854 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:17.839858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:17.839860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:17.840324 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:17.840524 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:17.840860 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 22:26:17.841143 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260619 22:26:17.843963 9959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d342a06e5ff46308e5c85b07b043514" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42985 } }
I20260619 22:26:17.844270 9959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:17.844364 9959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d342a06e5ff46308e5c85b07b043514, State: Initialized, Role: FOLLOWER
I20260619 22:26:17.844902 9959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d342a06e5ff46308e5c85b07b043514" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42985 } }
I20260619 22:26:17.845041 9959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:17.845141 9959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:17.845275 9959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:17.846406 9959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d342a06e5ff46308e5c85b07b043514" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42985 } }
I20260619 22:26:17.846814 9959 leader_election.cc:302] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d342a06e5ff46308e5c85b07b043514; no voters:
I20260619 22:26:17.847213 9959 leader_election.cc:288] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:17.847338 9964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:17.847579 9964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [term 1 LEADER]: Becoming Leader. State: Replica: 3d342a06e5ff46308e5c85b07b043514, State: Running, Role: LEADER
I20260619 22:26:17.847898 9964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d342a06e5ff46308e5c85b07b043514" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42985 } }
I20260619 22:26:17.848556 9959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:17.849644 9966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d342a06e5ff46308e5c85b07b043514. Latest consensus state: current_term: 1 leader_uuid: "3d342a06e5ff46308e5c85b07b043514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d342a06e5ff46308e5c85b07b043514" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42985 } } }
I20260619 22:26:17.849810 9966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:17.849644 9965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d342a06e5ff46308e5c85b07b043514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d342a06e5ff46308e5c85b07b043514" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 42985 } } }
I20260619 22:26:17.849977 9965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:17.850301 9973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:17.853797 9973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:17.859668 9973 catalog_manager.cc:1347] Generated new cluster ID: 27e864438fce44949c988a07975ad051
I20260619 22:26:17.859774 9973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:17.885214 9973 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:17.886343 9973 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:17.893530 9973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514: Generated new TSK 0
I20260619 22:26:17.894366 9973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:17.927240 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42985
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:18.040082 9985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:18.040380 9985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:18.040426 9985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:18.044453 9985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:18.044519 9985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:18.044581 9985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:18.050331 9985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:18.051437 9985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:18.052412 9985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:18.062317 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:18.062317 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:18.062317 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:18.062614 9985 server_base.cc:1034] running on GCE node
I20260619 22:26:18.063266 9985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:18.063957 9985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:18.065166 9985 hybrid_clock.cc:648] HybridClock initialized: now 1781907978065149 us; error 32 us; skew 500 ppm
Jun 19 22:26:18 dist-test-slave-bt27 krb5kdc[9879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907978, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:18.068104 9985 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:18.068939 9985 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:18.069041 9985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:18.069283 9985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:18.071496 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "a9ca202fffa741859959fdd247a8f922"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "632e81b4f2b080eb2efc64d67d40619a"
server_key_iv: "e04b43d625b3a14bbaf373a75645a8fe"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.072089 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "a9ca202fffa741859959fdd247a8f922"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "632e81b4f2b080eb2efc64d67d40619a"
server_key_iv: "e04b43d625b3a14bbaf373a75645a8fe"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.076136 9985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:18.078835 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.080257 9985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:18.080406 9985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "a9ca202fffa741859959fdd247a8f922"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "632e81b4f2b080eb2efc64d67d40619a"
server_key_iv: "e04b43d625b3a14bbaf373a75645a8fe"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.080534 9985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:18.100572 9985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:18.104254 9985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:18.104499 9985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:18.105176 9985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:18.106081 9985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:18.106189 9985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.106251 9985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:18.106304 9985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.115495 9985 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:34981
I20260619 22:26:18.115514 10112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:34981 every 8 connection(s)
I20260619 22:26:18.116164 9985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:18.124155 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 9985
I20260619 22:26:18.124298 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:18.124655 5827 external_mini_cluster.cc:1428] Setting key 4904ab9ed89aaac104d64efc576a4bb0
Jun 19 22:26:18 dist-test-slave-bt27 krb5kdc[9879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907978, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:18.130759 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.118037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:39981 (local address 127.5.176.254:42985)
0619 22:26:18.118222 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.118228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.119586 (+ 1358us) server_negotiation.cc:407] Connection header received
0619 22:26:18.120430 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.120433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.120480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.120559 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:18.121813 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.122463 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.123602 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.123895 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.123986 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.127008 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:18.127011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:18.127013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:18.127037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:18.128575 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.129116 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.129120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.129122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.129213 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.129562 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.129567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.129569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.129758 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:18.129875 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.130454 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.130586 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:26:18.131584 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.118527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42985 (local address 127.5.176.193:39981)
0619 22:26:18.119420 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.119466 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.120236 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.120808 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.120816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.121171 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:18.121631 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.121644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.122583 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.122585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.123470 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.123478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.124081 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.124086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.124340 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.125068 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:18.125093 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:18.126834 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:18.128703 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:18.128709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:18.128726 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:18.129011 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:18.129312 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:18.129316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:18.129318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:18.129466 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:18.129871 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:18.129879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:18.130180 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.130392 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.130731 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":110,"threads_started":1}
I20260619 22:26:18.132896 10113 heartbeater.cc:347] Connected to a master server at 127.5.176.254:42985
I20260619 22:26:18.133169 10113 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:18.133725 10113 heartbeater.cc:511] Master 127.5.176.254:42985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:26:18.135393 9923 ts_manager.cc:194] Registered new tserver with Master: a9ca202fffa741859959fdd247a8f922 (127.5.176.193:34981)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:18.136983 9923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:39981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:18.152699 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42985
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:18.266883 10118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:18.267148 10118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:18.267194 10118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:18.271356 10118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:18.271420 10118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:18.271512 10118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:18.276849 10118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:18.277940 10118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:18.279080 10118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:18.288744 10126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:18.288865 10123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:18.288745 10124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:18.289103 10118 server_base.cc:1034] running on GCE node
I20260619 22:26:18.289532 10118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:18.290238 10118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:18.291443 10118 hybrid_clock.cc:648] HybridClock initialized: now 1781907978291412 us; error 51 us; skew 500 ppm
Jun 19 22:26:18 dist-test-slave-bt27 krb5kdc[9879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907978, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:18.294394 10118 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:18.295135 10118 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:18.295202 10118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:18.295364 10118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:18.297408 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "324f52b9919b47b3809a2eb2b85d581e"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "4b90a6fede3fd93689199fff88cc6c22"
server_key_iv: "528b02c7c07b3bae055b5a3d9613869a"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.297941 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "324f52b9919b47b3809a2eb2b85d581e"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "4b90a6fede3fd93689199fff88cc6c22"
server_key_iv: "528b02c7c07b3bae055b5a3d9613869a"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.301949 10118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:26:18.304512 10132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.305866 10118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:18.305968 10118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "324f52b9919b47b3809a2eb2b85d581e"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "4b90a6fede3fd93689199fff88cc6c22"
server_key_iv: "528b02c7c07b3bae055b5a3d9613869a"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.306119 10118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:18.327111 10118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:18.330459 10118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:18.330642 10118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:18.331265 10118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:18.332073 10118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:18.332121 10118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.332159 10118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:18.332175 10118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.341727 10118 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:39561
I20260619 22:26:18.341741 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:39561 every 8 connection(s)
I20260619 22:26:18.342408 10118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:18.350054 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10118
I20260619 22:26:18.350229 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:18.350575 5827 external_mini_cluster.cc:1428] Setting key 61ba8cd4f415f31ca333b5d5a2e64608
Jun 19 22:26:18 dist-test-slave-bt27 krb5kdc[9879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907978, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:18.357003 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.344158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:58603 (local address 127.5.176.254:42985)
0619 22:26:18.344316 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.344320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.345407 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 22:26:18.346343 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.346347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.346406 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.346514 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:18.347678 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.348270 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.349565 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.349852 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.349968 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.352958 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:18.352961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:18.352963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:18.352986 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:18.354785 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.355295 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.355301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.355306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.355396 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.355702 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.355707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.355712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.355961 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:18.356069 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.356615 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.356752 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:18.357715 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.344462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42985 (local address 127.5.176.194:58603)
0619 22:26:18.345254 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.345294 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.346085 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.346686 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.346693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.347068 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:18.347534 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.347546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.348387 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.348392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.349435 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.349444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.350005 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.350009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.350258 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.350996 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:18.351013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:18.352762 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:18.354918 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:18.354923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:18.354935 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:18.355185 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:18.355487 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:18.355489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:18.355491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:18.355614 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:18.356065 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:18.356070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:18.356371 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.356530 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.356795 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":103,"threads_started":1}
I20260619 22:26:18.359105 10246 heartbeater.cc:347] Connected to a master server at 127.5.176.254:42985
I20260619 22:26:18.359385 10246 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:18.359953 10246 heartbeater.cc:511] Master 127.5.176.254:42985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:26:18.361140 9923 ts_manager.cc:194] Registered new tserver with Master: 324f52b9919b47b3809a2eb2b85d581e (127.5.176.194:39561)
I20260619 22:26:18.361723 9923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:58603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:18.379565 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42985
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:18.489841 10251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:18.490091 10251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:18.490185 10251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:18.494213 10251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:18.494269 10251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:18.494329 10251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:18.499504 10251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:18.500492 10251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:18.501576 10251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:18.511116 10259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:18.511097 10257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:18.511219 10256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:18.511552 10251 server_base.cc:1034] running on GCE node
I20260619 22:26:18.512007 10251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:18.512662 10251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:18.513864 10251 hybrid_clock.cc:648] HybridClock initialized: now 1781907978513851 us; error 33 us; skew 500 ppm
Jun 19 22:26:18 dist-test-slave-bt27 krb5kdc[9879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907978, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:18.516750 10251 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:18.517503 10251 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:18.517563 10251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:18.517726 10251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:18.519765 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "4840fd4722f742dd8af104aebf717196"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "80a53b9c07ddaaf4031dd2244e8d601d"
server_key_iv: "55953e68536d9bfc7cccddb5b6e202a0"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.520278 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "4840fd4722f742dd8af104aebf717196"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "80a53b9c07ddaaf4031dd2244e8d601d"
server_key_iv: "55953e68536d9bfc7cccddb5b6e202a0"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.523960 10251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:26:18.526361 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.527547 10251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:18.527649 10251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "4840fd4722f742dd8af104aebf717196"
format_stamp: "Formatted at 2026-06-19 22:26:18 on dist-test-slave-bt27"
server_key: "80a53b9c07ddaaf4031dd2244e8d601d"
server_key_iv: "55953e68536d9bfc7cccddb5b6e202a0"
server_key_version: "encryptionkey@0"
I20260619 22:26:18.527732 10251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:18.540511 10251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:18.543789 10251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:18.543949 10251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:18.544504 10251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:18.545362 10251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:18.545408 10251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.545449 10251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:18.545464 10251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:18.554889 10251 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:33033
I20260619 22:26:18.554970 10378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:33033 every 8 connection(s)
I20260619 22:26:18.555433 10251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:18.556452 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10251
I20260619 22:26:18.556569 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:18.556895 5827 external_mini_cluster.cc:1428] Setting key aa8f11b62df780de2937f80e64a74a37
Jun 19 22:26:18 dist-test-slave-bt27 krb5kdc[9879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907978, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:18.569795 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.557233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:59023 (local address 127.5.176.254:42985)
0619 22:26:18.557405 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.557410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.559086 (+ 1676us) server_negotiation.cc:407] Connection header received
0619 22:26:18.560009 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.560011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.560055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.560123 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:18.561345 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.561885 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.563171 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.563423 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.563515 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.566022 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:18.566024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:18.566027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:18.566051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:18.567749 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.568238 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.568240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.568241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.568311 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.568640 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.568642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.568643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.568848 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:18.568924 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.569448 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.569559 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:26:18.570837 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.557701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:42985 (local address 127.5.176.195:59023)
0619 22:26:18.558832 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.558891 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.559782 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.560324 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.560331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.560707 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:18.561195 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.561209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.562030 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.562033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.563043 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.563053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.563578 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.563582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.563740 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.564315 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:18.564331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:18.565862 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:18.567874 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:18.567878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:18.567890 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:18.568139 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:18.568408 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:18.568411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:18.568412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:18.568547 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:18.568945 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:18.568951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:18.569181 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.569394 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.569722 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":134,"threads_started":1}
I20260619 22:26:18.572048 10379 heartbeater.cc:347] Connected to a master server at 127.5.176.254:42985
I20260619 22:26:18.572327 10379 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:18.572794 10379 heartbeater.cc:511] Master 127.5.176.254:42985 requested a full tablet report, sending...
I20260619 22:26:18.573864 9923 ts_manager.cc:194] Registered new tserver with Master: 4840fd4722f742dd8af104aebf717196 (127.5.176.195:33033)
I20260619 22:26:18.574512 9923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:59023
I20260619 22:26:18.583166 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:26:18.592572 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.585269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.5.176.254:42985)
0619 22:26:18.585434 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.585438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.585539 (+ 101us) server_negotiation.cc:407] Connection header received
0619 22:26:18.585651 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.585653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.585694 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.585761 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:18.586628 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.587155 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.587946 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.588170 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.588270 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.589588 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:18.589591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:18.589593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:18.589618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:18.591104 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.591476 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.591479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.591480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.591551 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:18.591784 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:18.591786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:18.591787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:18.591995 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:18.592109 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.592281 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.592375 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:26:18.595858 9923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:18.598913 9923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:18.614418 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.608435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.5.176.194:39561)
0619 22:26:18.609603 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.609619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.609641 (+ 22us) server_negotiation.cc:407] Connection header received
0619 22:26:18.609708 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.609713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.609934 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.610071 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:18.610416 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.611129 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.613083 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.613628 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.613751 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.613911 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.614124 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.614243 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":109,"threads_started":1}
I20260619 22:26:18.614418 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.607777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34981 (local address 127.0.0.1:46984)
0619 22:26:18.608596 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.608652 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.608794 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.609648 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.609652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.609671 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:18.609968 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.610002 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.611407 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.611416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.612692 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.612701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.613968 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.613971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.614057 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.614076 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.614165 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.614225 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":121,"spinlock_wait_cycles":4608,"thread_start_us":101,"threads_started":1}
I20260619 22:26:18.614416 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.607988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:33033 (local address 127.0.0.1:59918)
0619 22:26:18.608614 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.608655 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.608793 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.609621 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.609627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.609652 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:18.609836 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.609843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.611373 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.611377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.612798 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.612807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.613805 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.613808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.613941 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.613971 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.614057 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.614217 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":105,"spinlock_wait_cycles":2816,"thread_start_us":104,"threads_started":1}
I20260619 22:26:18.614740 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.608810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.5.176.195:33033)
0619 22:26:18.609147 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.609156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.609195 (+ 39us) server_negotiation.cc:407] Connection header received
0619 22:26:18.609290 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.609293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.609452 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.609837 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:18.609976 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.611226 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.612901 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.613653 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.614306 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.614437 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.614489 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.614585 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260619 22:26:18.614847 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.608080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:39561 (local address 127.0.0.1:40806)
0619 22:26:18.608792 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.608804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.608881 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.610115 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.610121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.610158 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:18.610255 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.610266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.611256 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.611262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.612977 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.612984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.613750 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.613753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.613872 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.613970 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.614093 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.614225 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":191,"thread_start_us":91,"threads_started":1}
I20260619 22:26:18.615170 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.607940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.5.176.193:34981)
0619 22:26:18.608646 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.608655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.608956 (+ 301us) server_negotiation.cc:407] Connection header received
0619 22:26:18.609040 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.609045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.609295 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.609450 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:18.610090 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.611264 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.612819 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.613653 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.613769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.613910 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.614753 (+ 843us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.614863 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":109,"threads_started":1}
I20260619 22:26:18.617693 10180 tablet_service.cc:1467] Processing CreateTablet for tablet ddd6bcea92424c7b852e7e2eb7c1052c (DEFAULT_TABLE table=test-table [id=ab11fc8bcdf34ba4835b16c9f6c5f981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:18.617709 10047 tablet_service.cc:1467] Processing CreateTablet for tablet ddd6bcea92424c7b852e7e2eb7c1052c (DEFAULT_TABLE table=test-table [id=ab11fc8bcdf34ba4835b16c9f6c5f981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:18.617750 10313 tablet_service.cc:1467] Processing CreateTablet for tablet ddd6bcea92424c7b852e7e2eb7c1052c (DEFAULT_TABLE table=test-table [id=ab11fc8bcdf34ba4835b16c9f6c5f981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:18.618712 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd6bcea92424c7b852e7e2eb7c1052c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:18.618734 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd6bcea92424c7b852e7e2eb7c1052c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:18.618739 10313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd6bcea92424c7b852e7e2eb7c1052c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:18.625258 10396 tablet_bootstrap.cc:492] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Bootstrap starting.
I20260619 22:26:18.626633 10397 tablet_bootstrap.cc:492] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: Bootstrap starting.
I20260619 22:26:18.626633 10398 tablet_bootstrap.cc:492] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Bootstrap starting.
I20260619 22:26:18.627503 10396 tablet_bootstrap.cc:654] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:18.628567 10396 log.cc:824] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:18.628746 10397 tablet_bootstrap.cc:654] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:18.628746 10398 tablet_bootstrap.cc:654] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:18.629881 10397 log.cc:824] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:18.629881 10398 log.cc:824] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:18.630683 10396 tablet_bootstrap.cc:492] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: No bootstrap required, opened a new log
I20260619 22:26:18.630882 10396 ts_tablet_manager.cc:1397] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:18.631857 10398 tablet_bootstrap.cc:492] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: No bootstrap required, opened a new log
I20260619 22:26:18.632046 10398 ts_tablet_manager.cc:1397] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:26:18.634820 10396 raft_consensus.cc:348] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.635111 10396 raft_consensus.cc:374] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:18.635198 10396 raft_consensus.cc:729] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324f52b9919b47b3809a2eb2b85d581e, State: Initialized, Role: FOLLOWER
I20260619 22:26:18.635411 10397 tablet_bootstrap.cc:492] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: No bootstrap required, opened a new log
I20260619 22:26:18.635609 10397 ts_tablet_manager.cc:1397] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 22:26:18.635704 10396 consensus_queue.cc:260] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.635869 10398 raft_consensus.cc:348] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.636116 10398 raft_consensus.cc:374] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:18.636160 10398 raft_consensus.cc:729] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4840fd4722f742dd8af104aebf717196, State: Initialized, Role: FOLLOWER
I20260619 22:26:18.636385 10246 heartbeater.cc:503] Master 127.5.176.254:42985 was elected leader, sending a full tablet report...
I20260619 22:26:18.636745 10396 ts_tablet_manager.cc:1428] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:18.637063 10398 consensus_queue.cc:260] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.637907 10379 heartbeater.cc:503] Master 127.5.176.254:42985 was elected leader, sending a full tablet report...
I20260619 22:26:18.637960 10398 ts_tablet_manager.cc:1428] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:18.640213 10397 raft_consensus.cc:348] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.640472 10397 raft_consensus.cc:374] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:18.640573 10397 raft_consensus.cc:729] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ca202fffa741859959fdd247a8f922, State: Initialized, Role: FOLLOWER
I20260619 22:26:18.641014 10397 consensus_queue.cc:260] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.641644 10113 heartbeater.cc:503] Master 127.5.176.254:42985 was elected leader, sending a full tablet report...
I20260619 22:26:18.641952 10397 ts_tablet_manager.cc:1428] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 22:26:18.806507 10380 tablet.cc:2369] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:18.843631 10247 tablet.cc:2369] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:18.867511 10114 tablet.cc:2369] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:18.886482 10403 raft_consensus.cc:482] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:18.886703 10403 raft_consensus.cc:504] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.887923 10403 leader_election.cc:288] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ca202fffa741859959fdd247a8f922 (127.5.176.193:34981), 324f52b9919b47b3809a2eb2b85d581e (127.5.176.194:39561)
I20260619 22:26:18.891945 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.888334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:49057 (local address 127.5.176.194:39561)
0619 22:26:18.888465 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.888470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.888820 (+ 350us) server_negotiation.cc:407] Connection header received
0619 22:26:18.888898 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.888901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.888947 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.889028 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:18.889317 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.889843 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.890950 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.891352 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.891483 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.891534 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.891732 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.891814 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:26:18.891964 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.888251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34981 (local address 127.5.176.195:47209)
0619 22:26:18.888356 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.888373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.888472 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.888840 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.888846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.888871 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:18.889031 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.889038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.889881 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.889887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.890816 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.890824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.891464 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.891467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.891575 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.891645 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.891740 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.891814 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":101}
I20260619 22:26:18.891968 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.888420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:47209 (local address 127.5.176.193:34981)
0619 22:26:18.888531 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.888537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.888553 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:18.888612 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.888615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.888664 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.888802 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:18.889149 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.889769 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.890945 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.891345 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.891470 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.891523 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.891779 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.891850 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 22:26:18.891984 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.888258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:39561 (local address 127.5.176.195:49057)
0619 22:26:18.888708 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:18.888723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:18.888820 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:18.889048 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:18.889050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:18.889062 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:18.889196 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.889204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.889960 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.889962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.890819 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:18.890828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.891484 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.891487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:18.891613 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.891630 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:26:18.891688 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 22:26:18.891791 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":124,"thread_start_us":127,"threads_started":1}
I20260619 22:26:18.892675 10067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd6bcea92424c7b852e7e2eb7c1052c" candidate_uuid: "4840fd4722f742dd8af104aebf717196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ca202fffa741859959fdd247a8f922" is_pre_election: true
I20260619 22:26:18.892694 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd6bcea92424c7b852e7e2eb7c1052c" candidate_uuid: "4840fd4722f742dd8af104aebf717196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324f52b9919b47b3809a2eb2b85d581e" is_pre_election: true
I20260619 22:26:18.892980 10067 raft_consensus.cc:2454] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4840fd4722f742dd8af104aebf717196 in term 0.
I20260619 22:26:18.892990 10200 raft_consensus.cc:2454] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4840fd4722f742dd8af104aebf717196 in term 0.
I20260619 22:26:18.893473 10266 leader_election.cc:302] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324f52b9919b47b3809a2eb2b85d581e, 4840fd4722f742dd8af104aebf717196; no voters:
I20260619 22:26:18.893744 10403 raft_consensus.cc:2781] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:18.893849 10403 raft_consensus.cc:482] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:18.893909 10403 raft_consensus.cc:3037] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:18.894953 10403 raft_consensus.cc:504] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.895356 10403 leader_election.cc:288] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [CANDIDATE]: Term 1 election: Requested vote from peers a9ca202fffa741859959fdd247a8f922 (127.5.176.193:34981), 324f52b9919b47b3809a2eb2b85d581e (127.5.176.194:39561)
I20260619 22:26:18.895718 10067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd6bcea92424c7b852e7e2eb7c1052c" candidate_uuid: "4840fd4722f742dd8af104aebf717196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ca202fffa741859959fdd247a8f922"
I20260619 22:26:18.895828 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd6bcea92424c7b852e7e2eb7c1052c" candidate_uuid: "4840fd4722f742dd8af104aebf717196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324f52b9919b47b3809a2eb2b85d581e"
I20260619 22:26:18.895867 10067 raft_consensus.cc:3037] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:18.895990 10200 raft_consensus.cc:3037] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:18.897014 10067 raft_consensus.cc:2454] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4840fd4722f742dd8af104aebf717196 in term 1.
I20260619 22:26:18.897008 10200 raft_consensus.cc:2454] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4840fd4722f742dd8af104aebf717196 in term 1.
I20260619 22:26:18.897310 10266 leader_election.cc:302] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324f52b9919b47b3809a2eb2b85d581e, 4840fd4722f742dd8af104aebf717196; no voters:
I20260619 22:26:18.897475 10403 raft_consensus.cc:2781] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:18.897747 10403 raft_consensus.cc:686] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 1 LEADER]: Becoming Leader. State: Replica: 4840fd4722f742dd8af104aebf717196, State: Running, Role: LEADER
I20260619 22:26:18.898111 10403 consensus_queue.cc:237] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } }
I20260619 22:26:18.901578 9923 catalog_manager.cc:5495] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4840fd4722f742dd8af104aebf717196 (127.5.176.195). New cstate: current_term: 1 leader_uuid: "4840fd4722f742dd8af104aebf717196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4840fd4722f742dd8af104aebf717196" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:18.976948 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:18.973394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.5.176.195:33033)
0619 22:26:18.973527 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:18.973531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:18.973545 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:18.973672 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:18.973675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:18.973746 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:18.973856 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:18.974109 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.974643 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.975502 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:18.975744 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:18.975852 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:18.976158 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:18.976558 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:18.976685 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:18.976735 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 22:26:18.976789 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 22:26:18.982573 10067 raft_consensus.cc:1261] T ddd6bcea92424c7b852e7e2eb7c1052c P a9ca202fffa741859959fdd247a8f922 [term 1 FOLLOWER]: Refusing update from remote peer 4840fd4722f742dd8af104aebf717196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:18.982573 10200 raft_consensus.cc:1261] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 1 FOLLOWER]: Refusing update from remote peer 4840fd4722f742dd8af104aebf717196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:18.983273 10403 consensus_queue.cc:1035] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca202fffa741859959fdd247a8f922" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:18.983510 10406 consensus_queue.cc:1035] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f52b9919b47b3809a2eb2b85d581e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:18.991504 10411 mvcc.cc:205] Tried to move back new op lower bound from 7298695081905512448 to 7298695081569812480. Current Snapshot: MvccSnapshot[applied={T|T < 7298695081905512448}]
I20260619 22:26:18.991531 10409 mvcc.cc:205] Tried to move back new op lower bound from 7298695081905512448 to 7298695081569812480. Current Snapshot: MvccSnapshot[applied={T|T < 7298695081905512448}]
I20260619 22:26:19.000200 9923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:19.000404 9923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:19.003113 9923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d342a06e5ff46308e5c85b07b043514: Sending DeleteTablet for 3 replicas of tablet ddd6bcea92424c7b852e7e2eb7c1052c
I20260619 22:26:19.003947 10047 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd6bcea92424c7b852e7e2eb7c1052c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:18 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:46984
I20260619 22:26:19.003998 10313 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd6bcea92424c7b852e7e2eb7c1052c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:18 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:59918
I20260619 22:26:19.004007 10180 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd6bcea92424c7b852e7e2eb7c1052c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:18 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:40806
I20260619 22:26:19.004694 10420 tablet_replica.cc:330] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: stopping tablet replica
I20260619 22:26:19.005048 10420 raft_consensus.cc:2229] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:19.005242 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9985
I20260619 22:26:19.005630 10420 raft_consensus.cc:2258] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:19.005996 10421 tablet_replica.cc:330] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: stopping tablet replica
I20260619 22:26:19.006345 10421 raft_consensus.cc:2229] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:19.006645 10421 raft_consensus.cc:2258] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:19.009003 10420 ts_tablet_manager.cc:1905] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:19.011451 10420 ts_tablet_manager.cc:1918] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:19.011564 10420 log.cc:1196] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/ddd6bcea92424c7b852e7e2eb7c1052c
I20260619 22:26:19.011859 10420 ts_tablet_manager.cc:1939] T ddd6bcea92424c7b852e7e2eb7c1052c P 4840fd4722f742dd8af104aebf717196: Deleting consensus metadata
I20260619 22:26:19.013038 9909 catalog_manager.cc:4841] TS 4840fd4722f742dd8af104aebf717196 (127.5.176.195:33033): tablet ddd6bcea92424c7b852e7e2eb7c1052c (table test-table [id=ab11fc8bcdf34ba4835b16c9f6c5f981]) successfully deleted
W20260619 22:26:19.013738 9908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:34981 (error 108)
I20260619 22:26:19.014358 10421 ts_tablet_manager.cc:1905] T ddd6bcea92424c7b852e7e2eb7c1052c P 324f52b9919b47b3809a2eb2b85d581e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:19.014621 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10118
I20260619 22:26:19.015322 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.014537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34981 (local address 127.0.0.1:47000)
0619 22:26:19.015152 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.015221 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:34981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521}
W20260619 22:26:19.015491 9908 catalog_manager.cc:4584] TS a9ca202fffa741859959fdd247a8f922 (127.5.176.193:34981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddd6bcea92424c7b852e7e2eb7c1052c: Network error: Client connection negotiation failed: client connection to 127.5.176.193:34981: connect: Connection refused (error 111)
I20260619 22:26:19.022789 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10251
I20260619 22:26:19.024163 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.022978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:39561 (local address 127.0.0.1:40816)
0619 22:26:19.024024 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.024079 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":946}
I20260619 22:26:19.028996 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 9892
2026-06-19T22:26:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2548 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T22:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:19Z Disabled control of system clock
I20260619 22:26:19.059746 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43483
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 22:26:19.172000 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:19.172277 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:19.172320 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:19.176373 10426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:19.176441 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:19.176457 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:19.176474 10426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:19.176489 10426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:19.181285 10426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:43483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:19.182467 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:19.183897 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:19.192615 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:19.192615 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:19.192662 10426 server_base.cc:1034] running on GCE node
W20260619 22:26:19.192682 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:19.193360 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:19.194563 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:19.195765 10426 hybrid_clock.cc:648] HybridClock initialized: now 1781907979195743 us; error 41 us; skew 500 ppm
I20260619 22:26:19.200827 10426 webserver.cc:460] Webserver started at http://127.5.176.254:41445/ using document root <none> and password file <none>
I20260619 22:26:19.201463 10426 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:19.201562 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:19.201799 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:19.203982 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "04787d6a00104c6dac31ad62944f160a"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "d67a539645e5337e031a88a8dceaf84a"
server_key_iv: "19115af2cab7c80568740e284405359b"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.204578 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "04787d6a00104c6dac31ad62944f160a"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "d67a539645e5337e031a88a8dceaf84a"
server_key_iv: "19115af2cab7c80568740e284405359b"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.208367 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:19.211129 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.212466 10426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:19.212625 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "04787d6a00104c6dac31ad62944f160a"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "d67a539645e5337e031a88a8dceaf84a"
server_key_iv: "19115af2cab7c80568740e284405359b"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.212757 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:19.217470 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:19.218070 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:19.218288 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:19.225548 10426 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:43483
I20260619 22:26:19.225581 10492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:43483 every 8 connection(s)
I20260619 22:26:19.226554 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:19.229707 10493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:19.235512 10493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a: Bootstrap starting.
I20260619 22:26:19.235666 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10426
I20260619 22:26:19.235761 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:19.236054 5827 external_mini_cluster.cc:1428] Setting key fc5079bc6fcf19542930a282f6c0d260
I20260619 22:26:19.238379 10493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:19.239120 10493 log.cc:824] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:19.241590 10493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a: No bootstrap required, opened a new log
I20260619 22:26:19.242015 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.237478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.5.176.254:43483)
0619 22:26:19.238215 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.238231 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.238268 (+ 37us) server_negotiation.cc:407] Connection header received
0619 22:26:19.238941 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.238959 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.239369 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.239727 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.240493 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.240497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.240508 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.240518 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.240574 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.240589 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.240708 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.240818 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.241063 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260619 22:26:19.245936 10493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04787d6a00104c6dac31ad62944f160a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43483 } }
I20260619 22:26:19.246217 10493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:19.246276 10493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04787d6a00104c6dac31ad62944f160a, State: Initialized, Role: FOLLOWER
I20260619 22:26:19.246701 10493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04787d6a00104c6dac31ad62944f160a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43483 } }
I20260619 22:26:19.246824 10493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:19.246871 10493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:19.246943 10493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:19.247797 10493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04787d6a00104c6dac31ad62944f160a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43483 } }
I20260619 22:26:19.248087 10493 leader_election.cc:302] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04787d6a00104c6dac31ad62944f160a; no voters:
I20260619 22:26:19.248347 10493 leader_election.cc:288] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:19.248507 10498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:19.248760 10498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [term 1 LEADER]: Becoming Leader. State: Replica: 04787d6a00104c6dac31ad62944f160a, State: Running, Role: LEADER
I20260619 22:26:19.249118 10498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04787d6a00104c6dac31ad62944f160a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43483 } }
I20260619 22:26:19.249575 10493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:19.250932 10499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04787d6a00104c6dac31ad62944f160a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04787d6a00104c6dac31ad62944f160a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43483 } } }
I20260619 22:26:19.251051 10499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:19.251456 10508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:19.251497 10500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04787d6a00104c6dac31ad62944f160a. Latest consensus state: current_term: 1 leader_uuid: "04787d6a00104c6dac31ad62944f160a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04787d6a00104c6dac31ad62944f160a" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43483 } } }
I20260619 22:26:19.251597 10500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:19.254806 10508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:19.260013 10508 catalog_manager.cc:1347] Generated new cluster ID: 783e6f0d8f4d439d9a3507f070f1661a
I20260619 22:26:19.260082 10508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:19.284566 10508 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:19.284960 10508 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:19.292804 10508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a: Generated new TSK 0
I20260619 22:26:19.293555 10508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:19.297287 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43483
--builtin_ntp_servers=127.5.176.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 22:26:19.410524 10517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:19.410794 10517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:19.410833 10517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:19.414911 10517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:19.414969 10517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:19.415026 10517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:19.419999 10517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:19.421021 10517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:19.422387 10517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:19.432027 10522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:19.432029 10523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:19.432041 10525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:19.432154 10517 server_base.cc:1034] running on GCE node
I20260619 22:26:19.432775 10517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:19.433426 10517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:19.434619 10517 hybrid_clock.cc:648] HybridClock initialized: now 1781907979434598 us; error 32 us; skew 500 ppm
I20260619 22:26:19.436590 10517 webserver.cc:460] Webserver started at http://127.5.176.193:40927/ using document root <none> and password file <none>
I20260619 22:26:19.437194 10517 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:19.437265 10517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:19.437502 10517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:19.439637 10517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "7271567df68347bea73890fa66e1a1c7"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "04716f4d6ccce6032f4bef8cedbd995c"
server_key_iv: "acbd417692070090aead7e54f4749529"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.440215 10517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "7271567df68347bea73890fa66e1a1c7"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "04716f4d6ccce6032f4bef8cedbd995c"
server_key_iv: "acbd417692070090aead7e54f4749529"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.444093 10517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:19.446663 10531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.448010 10517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:19.448167 10517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "7271567df68347bea73890fa66e1a1c7"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "04716f4d6ccce6032f4bef8cedbd995c"
server_key_iv: "acbd417692070090aead7e54f4749529"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.448299 10517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:19.453137 10517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:19.453744 10517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:19.453882 10517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:19.454398 10517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:19.455296 10517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:19.455341 10517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.455379 10517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:19.455430 10517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.464556 10517 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:42041
I20260619 22:26:19.464627 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:42041 every 8 connection(s)
I20260619 22:26:19.465518 10517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:19.471926 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.467474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:37547 (local address 127.5.176.254:43483)
0619 22:26:19.467637 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.467642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.468821 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 22:26:19.470122 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.470125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.470201 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.470324 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.471440 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.471444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.471448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.471467 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.471487 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.471495 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.471594 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.471739 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.471787 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:26:19.472821 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.467939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43483 (local address 127.5.176.193:37547)
0619 22:26:19.468631 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.468674 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:19.469895 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:19.470498 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:19.470504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:19.470592 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:19.471181 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:19.471183 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:19.471224 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:19.471291 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:19.471577 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:19.471586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:19.471592 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:26:19.471814 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 22:26:19.472054 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260619 22:26:19.473584 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10517
I20260619 22:26:19.473721 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:19.474080 5827 external_mini_cluster.cc:1428] Setting key 2e5b456746e6cc290561c5a6c797b376
I20260619 22:26:19.474550 10645 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43483
I20260619 22:26:19.474809 10645 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:19.475302 10645 heartbeater.cc:511] Master 127.5.176.254:43483 requested a full tablet report, sending...
I20260619 22:26:19.476307 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43483
--builtin_ntp_servers=127.5.176.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 22:26:19.477027 10457 ts_manager.cc:194] Registered new tserver with Master: 7271567df68347bea73890fa66e1a1c7 (127.5.176.193:42041)
W20260619 22:26:19.591465 10648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:19.591741 10648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:19.591785 10648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:19.595839 10648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:19.595896 10648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:19.595957 10648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:19.600889 10648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:19.602041 10648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:19.603578 10648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:19.613472 10654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:19.613487 10653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:19.613543 10648 server_base.cc:1034] running on GCE node
W20260619 22:26:19.613487 10656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:19.614212 10648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:19.614827 10648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:19.616036 10648 hybrid_clock.cc:648] HybridClock initialized: now 1781907979616005 us; error 46 us; skew 500 ppm
I20260619 22:26:19.617890 10648 webserver.cc:460] Webserver started at http://127.5.176.194:36089/ using document root <none> and password file <none>
I20260619 22:26:19.618541 10648 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:19.618631 10648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:19.618860 10648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:19.621026 10648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "6a85d7a9352b4762ade143c4f27fa25e"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "a81c622c38c948ecd3e60e5c9ef2f208"
server_key_iv: "3076c123dd1c21c38154073584584b2a"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.621604 10648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "6a85d7a9352b4762ade143c4f27fa25e"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "a81c622c38c948ecd3e60e5c9ef2f208"
server_key_iv: "3076c123dd1c21c38154073584584b2a"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.625653 10648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:26:19.628490 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.629899 10648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:19.630012 10648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "6a85d7a9352b4762ade143c4f27fa25e"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "a81c622c38c948ecd3e60e5c9ef2f208"
server_key_iv: "3076c123dd1c21c38154073584584b2a"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.630117 10648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:19.634985 10648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:19.635586 10648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:19.635742 10648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:19.636266 10648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:19.637254 10648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:19.637306 10648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.637348 10648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:19.637367 10648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.648867 10648 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:43381
I20260619 22:26:19.648885 10775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:43381 every 8 connection(s)
I20260619 22:26:19.649847 10648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:19.653939 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10648
I20260619 22:26:19.654044 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:19.654399 5827 external_mini_cluster.cc:1428] Setting key 8236480612e362c6f9cc2476b4d8d822
I20260619 22:26:19.656546 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43483
--builtin_ntp_servers=127.5.176.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 22:26:19.656684 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.651921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:44563 (local address 127.5.176.254:43483)
0619 22:26:19.652061 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.652067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.653287 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 22:26:19.654784 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.654787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.654904 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.655038 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.656218 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.656221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.656223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.656235 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.656256 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.656264 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.656338 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.656513 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.656551 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:26:19.657796 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.652466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43483 (local address 127.5.176.194:44563)
0619 22:26:19.653138 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.653178 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:19.654523 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:19.655206 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:19.655213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:19.655300 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:19.655928 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:19.655930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:19.655982 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:19.656069 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:19.656354 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:19.656365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:19.656374 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:26:19.656553 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 22:26:19.656802 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260619 22:26:19.658984 10776 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43483
I20260619 22:26:19.659229 10776 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:19.659690 10776 heartbeater.cc:511] Master 127.5.176.254:43483 requested a full tablet report, sending...
I20260619 22:26:19.660745 10457 ts_manager.cc:194] Registered new tserver with Master: 6a85d7a9352b4762ade143c4f27fa25e (127.5.176.194:43381)
W20260619 22:26:19.768906 10779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:19.769181 10779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:19.769222 10779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:19.773380 10779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:19.773442 10779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:19.773500 10779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:19.778385 10779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:19.779484 10779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:19.780870 10779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:19.790602 10784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:19.790606 10787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:19.790606 10785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:19.790998 10779 server_base.cc:1034] running on GCE node
I20260619 22:26:19.791451 10779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:19.792125 10779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:19.793292 10779 hybrid_clock.cc:648] HybridClock initialized: now 1781907979793272 us; error 30 us; skew 500 ppm
I20260619 22:26:19.795172 10779 webserver.cc:460] Webserver started at http://127.5.176.195:36735/ using document root <none> and password file <none>
I20260619 22:26:19.795773 10779 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:19.795835 10779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:19.796043 10779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:19.798064 10779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "fad7edb28e0442bd85d8ca2c163f333c"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "1c78f4695b39fecce7a72de1d238a417"
server_key_iv: "96cc7919505477f1b793a62ded34e33e"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.798673 10779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "fad7edb28e0442bd85d8ca2c163f333c"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "1c78f4695b39fecce7a72de1d238a417"
server_key_iv: "96cc7919505477f1b793a62ded34e33e"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.802304 10779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:19.804618 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.805871 10779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:19.805967 10779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "fad7edb28e0442bd85d8ca2c163f333c"
format_stamp: "Formatted at 2026-06-19 22:26:19 on dist-test-slave-bt27"
server_key: "1c78f4695b39fecce7a72de1d238a417"
server_key_iv: "96cc7919505477f1b793a62ded34e33e"
server_key_version: "encryptionkey@0"
I20260619 22:26:19.806102 10779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:19.810699 10779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:19.811298 10779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:19.811471 10779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:19.812005 10779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:19.812882 10779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:19.812927 10779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.812990 10779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:19.813024 10779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:19.821745 10779 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:35471
I20260619 22:26:19.821784 10906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:35471 every 8 connection(s)
I20260619 22:26:19.822686 10779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:19.824115 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10779
I20260619 22:26:19.824249 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:19.824544 5827 external_mini_cluster.cc:1428] Setting key 3652de437113d4e6cd8d07cbf8128e3d
I20260619 22:26:19.829473 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.824866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:56065 (local address 127.5.176.254:43483)
0619 22:26:19.825037 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.825041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.826325 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 22:26:19.827803 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.827806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.827854 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.827928 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.829078 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.829082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.829084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.829095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.829114 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.829123 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.829204 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.829323 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.829363 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:26:19.830506 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.825301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43483 (local address 127.5.176.195:56065)
0619 22:26:19.826188 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.826239 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:19.827577 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:19.828103 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:19.828110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:19.828204 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:19.828811 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:19.828813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:19.828864 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:19.828935 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:19.829201 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:19.829207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:19.829211 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 22:26:19.829349 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 22:26:19.829595 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":152,"threads_started":1}
I20260619 22:26:19.831715 10907 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43483
I20260619 22:26:19.831983 10907 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:19.832427 10907 heartbeater.cc:511] Master 127.5.176.254:43483 requested a full tablet report, sending...
I20260619 22:26:19.833593 10457 ts_manager.cc:194] Registered new tserver with Master: fad7edb28e0442bd85d8ca2c163f333c (127.5.176.195:35471)
I20260619 22:26:19.838749 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:26:19.842659 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.840907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.5.176.254:43483)
0619 22:26:19.841089 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.841094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.841219 (+ 125us) server_negotiation.cc:407] Connection header received
0619 22:26:19.841382 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.841386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.841430 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.841514 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.842230 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.842234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.842237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.842249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.842283 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.842292 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.842384 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.842467 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.842508 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 22:26:19.846007 10457 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:19.849048 10457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:19.861279 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.858397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:35471 (local address 127.0.0.1:47722)
0619 22:26:19.858800 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.858857 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:19.858987 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:19.859743 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:19.859750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:19.859839 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:19.860440 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:19.860442 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:19.860464 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:19.860533 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:19.860918 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:19.860929 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:19.860938 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:26:19.861048 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 22:26:19.861116 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260619 22:26:19.861650 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.858552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.5.176.195:35471)
0619 22:26:19.859264 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.859281 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.859314 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:26:19.859371 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.859377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.859594 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.859742 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.860680 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.860687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.860699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.860713 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.860770 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.860809 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.860928 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.861346 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.861431 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":124,"threads_started":1}
I20260619 22:26:19.862310 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.858706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43381 (local address 127.0.0.1:39010)
0619 22:26:19.859112 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.859127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:19.859196 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:19.861012 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:19.861015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:19.861031 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:19.861669 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:19.861672 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:19.861683 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:19.861715 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:19.862048 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:19.862051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:19.862054 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 22:26:19.862158 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 22:26:19.862195 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260619 22:26:19.862805 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.859186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42041 (local address 127.0.0.1:58432)
0619 22:26:19.859416 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:19.859428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:19.859491 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:19.861126 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:19.861129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:19.861145 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:19.862132 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:19.862134 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:19.862144 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:19.862197 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:19.862573 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:19.862577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:19.862580 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 22:26:19.862659 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 22:26:19.862692 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260619 22:26:19.862977 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.859598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.5.176.193:42041)
0619 22:26:19.860073 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.860088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.860114 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:26:19.860166 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.860172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.860341 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.860483 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.862330 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.862337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.862350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.862361 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.862403 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.862436 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.862530 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.862702 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.862788 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":139,"threads_started":1}
I20260619 22:26:19.863363 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:19.859257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.5.176.194:43381)
0619 22:26:19.860577 (+ 1320us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:19.860589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:19.860613 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:26:19.860666 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:19.860671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:19.860826 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:19.861011 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:19.861867 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:19.861874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:19.861884 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:19.861893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:19.861926 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:19.861953 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:19.862036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:19.862197 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 22:26:19.862270 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1212,"thread_start_us":158,"threads_started":1}
I20260619 22:26:19.864873 10841 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff4f17afaa345dab7af053c2ba4b108 (DEFAULT_TABLE table=test-table [id=e14a30dcc6d4412f933f0e2ab145bd78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:19.865854 10841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff4f17afaa345dab7af053c2ba4b108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:19.865557 10579 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff4f17afaa345dab7af053c2ba4b108 (DEFAULT_TABLE table=test-table [id=e14a30dcc6d4412f933f0e2ab145bd78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:19.866613 10579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff4f17afaa345dab7af053c2ba4b108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:19.866225 10710 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff4f17afaa345dab7af053c2ba4b108 (DEFAULT_TABLE table=test-table [id=e14a30dcc6d4412f933f0e2ab145bd78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:19.867177 10710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff4f17afaa345dab7af053c2ba4b108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:19.872205 10924 tablet_bootstrap.cc:492] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Bootstrap starting.
I20260619 22:26:19.874271 10925 tablet_bootstrap.cc:492] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Bootstrap starting.
I20260619 22:26:19.874680 10926 tablet_bootstrap.cc:492] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: Bootstrap starting.
I20260619 22:26:19.874976 10924 tablet_bootstrap.cc:654] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:19.876163 10924 log.cc:824] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:19.876708 10926 tablet_bootstrap.cc:654] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:19.876710 10925 tablet_bootstrap.cc:654] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:19.877574 10926 log.cc:824] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:19.877574 10925 log.cc:824] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:19.878412 10924 tablet_bootstrap.cc:492] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: No bootstrap required, opened a new log
I20260619 22:26:19.878623 10924 ts_tablet_manager.cc:1397] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:26:19.879652 10926 tablet_bootstrap.cc:492] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: No bootstrap required, opened a new log
I20260619 22:26:19.879652 10925 tablet_bootstrap.cc:492] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: No bootstrap required, opened a new log
I20260619 22:26:19.879845 10925 ts_tablet_manager.cc:1397] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:26:19.879845 10926 ts_tablet_manager.cc:1397] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:26:19.883174 10924 raft_consensus.cc:348] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:19.883467 10924 raft_consensus.cc:374] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:19.883533 10924 raft_consensus.cc:729] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad7edb28e0442bd85d8ca2c163f333c, State: Initialized, Role: FOLLOWER
I20260619 22:26:19.884007 10924 consensus_queue.cc:260] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:19.884521 10926 raft_consensus.cc:348] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:19.884743 10907 heartbeater.cc:503] Master 127.5.176.254:43483 was elected leader, sending a full tablet report...
I20260619 22:26:19.884793 10926 raft_consensus.cc:374] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:19.884847 10926 raft_consensus.cc:729] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7271567df68347bea73890fa66e1a1c7, State: Initialized, Role: FOLLOWER
I20260619 22:26:19.885138 10924 ts_tablet_manager.cc:1428] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:19.885300 10926 consensus_queue.cc:260] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:19.884521 10925 raft_consensus.cc:348] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:19.885481 10925 raft_consensus.cc:374] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:19.885550 10925 raft_consensus.cc:729] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a85d7a9352b4762ade143c4f27fa25e, State: Initialized, Role: FOLLOWER
I20260619 22:26:19.885995 10925 consensus_queue.cc:260] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:19.886118 10645 heartbeater.cc:503] Master 127.5.176.254:43483 was elected leader, sending a full tablet report...
I20260619 22:26:19.886303 10926 ts_tablet_manager.cc:1428] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 22:26:19.888406 10776 heartbeater.cc:503] Master 127.5.176.254:43483 was elected leader, sending a full tablet report...
I20260619 22:26:19.888460 10925 ts_tablet_manager.cc:1428] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260619 22:26:19.900972 10777 tablet.cc:2369] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:19.966777 10646 tablet.cc:2369] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:20.012717 10932 raft_consensus.cc:482] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:20.013190 10932 raft_consensus.cc:504] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:20.014618 10932 leader_election.cc:288] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fad7edb28e0442bd85d8ca2c163f333c (127.5.176.195:35471), 7271567df68347bea73890fa66e1a1c7 (127.5.176.193:42041)
I20260619 22:26:20.017048 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.015111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:35471 (local address 127.5.176.194:56997)
0619 22:26:20.015266 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:20.015282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:20.015378 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:20.015778 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:20.015781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:20.015799 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:20.016359 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:20.016361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:20.016374 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:20.016404 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:20.016711 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:20.016714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:20.016717 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 22:26:20.016876 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 22:26:20.016916 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 22:26:20.017048 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.015240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:56997 (local address 127.5.176.195:35471)
0619 22:26:20.015410 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.015417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.015437 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:26:20.015488 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.015499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.015546 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.015655 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:20.016560 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:20.016564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:20.016566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:20.016602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:20.016625 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:20.016635 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:20.016727 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:20.016888 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 22:26:20.016928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 22:26:20.017920 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.015421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42041 (local address 127.5.176.194:36721)
0619 22:26:20.015866 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:20.015878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:20.015954 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:20.016268 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:20.016270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:20.016287 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:20.017228 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:20.017232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:20.017244 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:20.017271 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:20.017651 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:20.017655 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:20.017658 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 22:26:20.017735 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 22:26:20.017769 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":134,"threads_started":1}
I20260619 22:26:20.017985 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.015571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:36721 (local address 127.5.176.193:42041)
0619 22:26:20.015722 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.015728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.015964 (+ 236us) server_negotiation.cc:407] Connection header received
0619 22:26:20.016096 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.016100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.016143 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.016229 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:20.017427 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:20.017431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:20.017434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:20.017448 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:20.017472 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:20.017485 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:20.017581 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:20.017812 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 22:26:20.017863 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:26:20.018172 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4f17afaa345dab7af053c2ba4b108" candidate_uuid: "6a85d7a9352b4762ade143c4f27fa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad7edb28e0442bd85d8ca2c163f333c" is_pre_election: true
I20260619 22:26:20.018489 10861 raft_consensus.cc:2454] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a85d7a9352b4762ade143c4f27fa25e in term 0.
I20260619 22:26:20.019022 10666 leader_election.cc:302] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a85d7a9352b4762ade143c4f27fa25e, fad7edb28e0442bd85d8ca2c163f333c; no voters:
I20260619 22:26:20.019342 10932 raft_consensus.cc:2781] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:20.019337 10599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4f17afaa345dab7af053c2ba4b108" candidate_uuid: "6a85d7a9352b4762ade143c4f27fa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7271567df68347bea73890fa66e1a1c7" is_pre_election: true
I20260619 22:26:20.019450 10932 raft_consensus.cc:482] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:20.019500 10932 raft_consensus.cc:3037] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:20.019651 10599 raft_consensus.cc:2454] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a85d7a9352b4762ade143c4f27fa25e in term 0.
I20260619 22:26:20.020758 10932 raft_consensus.cc:504] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:20.021088 10932 leader_election.cc:288] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [CANDIDATE]: Term 1 election: Requested vote from peers fad7edb28e0442bd85d8ca2c163f333c (127.5.176.195:35471), 7271567df68347bea73890fa66e1a1c7 (127.5.176.193:42041)
I20260619 22:26:20.021543 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4f17afaa345dab7af053c2ba4b108" candidate_uuid: "6a85d7a9352b4762ade143c4f27fa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad7edb28e0442bd85d8ca2c163f333c"
I20260619 22:26:20.021647 10599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4f17afaa345dab7af053c2ba4b108" candidate_uuid: "6a85d7a9352b4762ade143c4f27fa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7271567df68347bea73890fa66e1a1c7"
I20260619 22:26:20.021709 10861 raft_consensus.cc:3037] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:20.021812 10599 raft_consensus.cc:3037] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:20.023154 10861 raft_consensus.cc:2454] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a85d7a9352b4762ade143c4f27fa25e in term 1.
I20260619 22:26:20.023160 10599 raft_consensus.cc:2454] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a85d7a9352b4762ade143c4f27fa25e in term 1.
I20260619 22:26:20.023545 10666 leader_election.cc:302] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a85d7a9352b4762ade143c4f27fa25e, 7271567df68347bea73890fa66e1a1c7; no voters:
I20260619 22:26:20.023744 10932 raft_consensus.cc:2781] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:20.023989 10932 raft_consensus.cc:686] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 1 LEADER]: Becoming Leader. State: Replica: 6a85d7a9352b4762ade143c4f27fa25e, State: Running, Role: LEADER
I20260619 22:26:20.024362 10932 consensus_queue.cc:237] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } }
I20260619 22:26:20.028419 10455 catalog_manager.cc:5495] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a85d7a9352b4762ade143c4f27fa25e (127.5.176.194). New cstate: current_term: 1 leader_uuid: "6a85d7a9352b4762ade143c4f27fa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a85d7a9352b4762ade143c4f27fa25e" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43381 } health_report { overall_health: HEALTHY } } }
I20260619 22:26:20.057374 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.055459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.5.176.194:43381)
0619 22:26:20.055601 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.055605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.055621 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:20.055702 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.055704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.055788 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.055865 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:20.056797 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:20.056801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:20.056804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:20.056823 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:20.056854 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:20.056865 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:20.057023 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:20.057205 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 22:26:20.057261 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:26:20.063949 10861 raft_consensus.cc:1261] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 1 FOLLOWER]: Refusing update from remote peer 6a85d7a9352b4762ade143c4f27fa25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:20.063949 10599 raft_consensus.cc:1261] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7 [term 1 FOLLOWER]: Refusing update from remote peer 6a85d7a9352b4762ade143c4f27fa25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:20.064761 10932 consensus_queue.cc:1035] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7271567df68347bea73890fa66e1a1c7" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:20.064985 10934 consensus_queue.cc:1035] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad7edb28e0442bd85d8ca2c163f333c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:20.072854 10939 mvcc.cc:205] Tried to move back new op lower bound from 7298695086333132800 to 7298695086182645760. Current Snapshot: MvccSnapshot[applied={T|T < 7298695086333132800}]
I20260619 22:26:20.081660 10457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:20.081875 10457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:20.084393 10457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04787d6a00104c6dac31ad62944f160a: Sending DeleteTablet for 3 replicas of tablet 1ff4f17afaa345dab7af053c2ba4b108
I20260619 22:26:20.085310 10710 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff4f17afaa345dab7af053c2ba4b108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:20 UTC) from {username='slave'} at 127.0.0.1:39010
I20260619 22:26:20.085482 10579 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff4f17afaa345dab7af053c2ba4b108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:20 UTC) from {username='slave'} at 127.0.0.1:58432
I20260619 22:26:20.085659 10841 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff4f17afaa345dab7af053c2ba4b108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:20 UTC) from {username='slave'} at 127.0.0.1:47722
I20260619 22:26:20.085826 10946 tablet_replica.cc:330] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: stopping tablet replica
I20260619 22:26:20.086110 10948 tablet_replica.cc:330] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: stopping tablet replica
I20260619 22:26:20.086167 10946 raft_consensus.cc:2229] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:20.086421 10948 raft_consensus.cc:2229] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:20.086732 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10517
I20260619 22:26:20.086735 10948 raft_consensus.cc:2258] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:20.086858 10947 tablet_replica.cc:330] T 1ff4f17afaa345dab7af053c2ba4b108 P 7271567df68347bea73890fa66e1a1c7: stopping tablet replica
I20260619 22:26:20.088299 10948 ts_tablet_manager.cc:1905] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:20.089417 10946 raft_consensus.cc:2258] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:20.091418 10948 ts_tablet_manager.cc:1918] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:20.091521 10948 log.cc:1196] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/1ff4f17afaa345dab7af053c2ba4b108
I20260619 22:26:20.091836 10948 ts_tablet_manager.cc:1939] T 1ff4f17afaa345dab7af053c2ba4b108 P fad7edb28e0442bd85d8ca2c163f333c: Deleting consensus metadata
I20260619 22:26:20.093068 10946 ts_tablet_manager.cc:1905] T 1ff4f17afaa345dab7af053c2ba4b108 P 6a85d7a9352b4762ade143c4f27fa25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:20.093191 10444 catalog_manager.cc:4841] TS fad7edb28e0442bd85d8ca2c163f333c (127.5.176.195:35471): tablet 1ff4f17afaa345dab7af053c2ba4b108 (table test-table [id=e14a30dcc6d4412f933f0e2ab145bd78]) successfully deleted
W20260619 22:26:20.094432 10444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:42041 (error 108)
I20260619 22:26:20.095293 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10648
I20260619 22:26:20.096762 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.096359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42041 (local address 127.0.0.1:58440)
0619 22:26:20.096563 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:20.096642 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260619 22:26:20.097048 10444 catalog_manager.cc:4584] TS 7271567df68347bea73890fa66e1a1c7 (127.5.176.193:42041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ff4f17afaa345dab7af053c2ba4b108: Network error: Client connection negotiation failed: client connection to 127.5.176.193:42041: connect: Connection refused (error 111)
I20260619 22:26:20.101024 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10779
I20260619 22:26:20.102416 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.101278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43381 (local address 127.0.0.1:39038)
0619 22:26:20.102221 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:20.102301 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":794}
I20260619 22:26:20.107057 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10426
2026-06-19T22:26:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1077 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 22:26:20.129647 5827 mini_oidc.cc:152] Starting JWKS server
I20260619 22:26:20.131579 5827 webserver.cc:460] Webserver started at https://127.0.0.1:43591/ using document root <none> and password file <none>
I20260619 22:26:20.131753 5827 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 22:26:20.132007 5827 webserver.cc:460] Webserver started at http://127.0.0.1:43839/ using document root <none> and password file <none>
2026-06-19T22:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:20Z Disabled control of system clock
I20260619 22:26:20.141651 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:41981
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:41981
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43591/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/testchainca.cert with env {}
W20260619 22:26:20.257259 10955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:20.257540 10955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:20.257586 10955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:20.261690 10955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:20.261768 10955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:20.261787 10955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:20.261809 10955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:20.261828 10955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:26:20.261848 10955 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:26:20.261864 10955 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43591/jwks.json
I20260619 22:26:20.267539 10955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:41981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43591/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:20.268587 10955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:20.269572 10955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:20.278095 10963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:20.278100 10960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:20.278185 10955 server_base.cc:1034] running on GCE node
W20260619 22:26:20.278095 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:20.278980 10955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:20.280092 10955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:20.281287 10955 hybrid_clock.cc:648] HybridClock initialized: now 1781907980281254 us; error 47 us; skew 500 ppm
I20260619 22:26:20.283192 10955 webserver.cc:460] Webserver started at http://127.5.176.254:38171/ using document root <none> and password file <none>
I20260619 22:26:20.283777 10955 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:20.283840 10955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:20.284055 10955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:20.286069 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "6e4d803b08374d14804141dc705d0672"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "3e9b4e93d9374e51af0f06a014cc7634"
server_key_iv: "d7191464235bbe50ffdb6b9a85e03855"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.286707 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "6e4d803b08374d14804141dc705d0672"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "3e9b4e93d9374e51af0f06a014cc7634"
server_key_iv: "d7191464235bbe50ffdb6b9a85e03855"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.290436 10955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:26:20.292790 10969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:20.294091 10955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:20.294260 10955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "6e4d803b08374d14804141dc705d0672"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "3e9b4e93d9374e51af0f06a014cc7634"
server_key_iv: "d7191464235bbe50ffdb6b9a85e03855"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.294373 10955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:20.322726 10955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:20.323463 10955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:20.323664 10955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:20.330444 10955 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:41981
I20260619 22:26:20.330456 11021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:41981 every 8 connection(s)
I20260619 22:26:20.331367 10955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:20.334599 11022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:20.338088 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 10955
I20260619 22:26:20.338261 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:20.338546 5827 external_mini_cluster.cc:1428] Setting key 14b164b9f31d647b85252c8a3ee65c1e
I20260619 22:26:20.340431 11022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672: Bootstrap starting.
I20260619 22:26:20.342754 11022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:20.343538 11022 log.cc:824] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:20.345624 11022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672: No bootstrap required, opened a new log
I20260619 22:26:20.347491 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.339896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.5.176.254:41981)
0619 22:26:20.340493 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.340508 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.340542 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:26:20.341194 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.341209 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.341587 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.341853 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:20.342241 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.343271 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.344356 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.344828 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.344942 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.345778 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:20.345783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:20.345795 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:20.346222 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:20.346285 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:20.346306 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:20.346381 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:20.346530 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 22:26:20.346806 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260619 22:26:20.349315 11022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4d803b08374d14804141dc705d0672" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41981 } }
I20260619 22:26:20.349592 11022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:20.349669 11022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4d803b08374d14804141dc705d0672, State: Initialized, Role: FOLLOWER
I20260619 22:26:20.350080 11022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4d803b08374d14804141dc705d0672" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41981 } }
I20260619 22:26:20.350198 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:26:20.350277 11022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:20.350363 11022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:20.350476 11022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:20.351490 11022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4d803b08374d14804141dc705d0672" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41981 } }
I20260619 22:26:20.351861 11022 leader_election.cc:302] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4d803b08374d14804141dc705d0672; no voters:
I20260619 22:26:20.352190 11022 leader_election.cc:288] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:20.352337 11027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:20.352636 11027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [term 1 LEADER]: Becoming Leader. State: Replica: 6e4d803b08374d14804141dc705d0672, State: Running, Role: LEADER
I20260619 22:26:20.352973 11027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4d803b08374d14804141dc705d0672" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41981 } }
I20260619 22:26:20.353421 11022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:20.354615 11029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e4d803b08374d14804141dc705d0672. Latest consensus state: current_term: 1 leader_uuid: "6e4d803b08374d14804141dc705d0672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4d803b08374d14804141dc705d0672" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41981 } } }
I20260619 22:26:20.354735 11029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:20.355003 11028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e4d803b08374d14804141dc705d0672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4d803b08374d14804141dc705d0672" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41981 } } }
I20260619 22:26:20.355121 11028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:20.355101 11036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:20.358109 11036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:20.364583 11036 catalog_manager.cc:1347] Generated new cluster ID: 0b63c90aa96c4f9b98926113d9f25a51
I20260619 22:26:20.364702 11036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:20.392830 11036 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:20.393883 11036 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:20.401280 11036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e4d803b08374d14804141dc705d0672: Generated new TSK 0
I20260619 22:26:20.402055 11036 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 22:26:20.432483 11052 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 22:26:20.432607 11052 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 22:26:20.433193 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.415236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.5.176.254:41981)
0619 22:26:20.415383 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.415388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.415624 (+ 236us) server_negotiation.cc:407] Connection header received
0619 22:26:20.415846 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.415848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.415897 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.415976 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:26:20.416265 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.416817 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.418046 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.418348 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.418429 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.418848 (+ 419us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:26:20.432793 (+ 13945us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 22:26:20.433024 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 22:26:20.433291 11025 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 22:26:20.434602 11051 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:20.415158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41981 (local address 127.0.0.1:48242)
0619 22:26:20.415526 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:20.415543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:20.415677 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:20.416025 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:20.416028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:20.416040 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:26:20.416161 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:20.416168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.416931 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.416935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:20.417921 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:20.417932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.418470 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.418474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:20.418569 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.418624 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:26:20.433021 (+ 14397us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 22:26:20.433281 (+ 260us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.176.254:41981: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":130,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.176.254:41981: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 22:26:20.435487 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 10955
2026-06-19T22:26:20Z chronyd exiting
I20260619 22:26:20.472525 5827 test_util.cc:175] -----------------------------------------------
I20260619 22:26:20.472638 5827 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781907951519384-5827-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 22:26:20.473680 5827 mini_oidc.cc:152] Starting JWKS server
I20260619 22:26:20.474614 5827 webserver.cc:460] Webserver started at https://127.0.0.1:39959/ using document root <none> and password file <none>
I20260619 22:26:20.474740 5827 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 22:26:20.475006 5827 webserver.cc:460] Webserver started at http://127.0.0.1:39769/ using document root <none> and password file <none>
2026-06-19T22:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:20Z Disabled control of system clock
I20260619 22:26:20.483729 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43325
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:43325
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39959/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/testchainca.cert with env {}
W20260619 22:26:20.595448 11058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:20.595739 11058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:20.595813 11058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:20.599915 11058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:20.599979 11058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:20.599995 11058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:20.600013 11058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:20.600028 11058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:26:20.600045 11058 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:26:20.600059 11058 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39959/jwks.json
I20260619 22:26:20.605602 11058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:43325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39959/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:20.606781 11058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:20.607760 11058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:20.616529 11064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:20.616606 11063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:20.616600 11058 server_base.cc:1034] running on GCE node
W20260619 22:26:20.616715 11066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:20.617360 11058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:20.618594 11058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:20.619802 11058 hybrid_clock.cc:648] HybridClock initialized: now 1781907980619780 us; error 37 us; skew 500 ppm
I20260619 22:26:20.621721 11058 webserver.cc:460] Webserver started at http://127.5.176.254:37353/ using document root <none> and password file <none>
I20260619 22:26:20.622421 11058 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:20.622516 11058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:20.622746 11058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:20.624934 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "c39d7062c1524a60a1232c1e991115b3"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "ffd84acb22ec1fe09efce9b6e8b03c9e"
server_key_iv: "d8b97a12c041cc0c8b64a66f99e55531"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.625509 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "c39d7062c1524a60a1232c1e991115b3"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "ffd84acb22ec1fe09efce9b6e8b03c9e"
server_key_iv: "d8b97a12c041cc0c8b64a66f99e55531"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.629287 11058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 22:26:20.631716 11072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:20.632926 11058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:20.633064 11058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "c39d7062c1524a60a1232c1e991115b3"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "ffd84acb22ec1fe09efce9b6e8b03c9e"
server_key_iv: "d8b97a12c041cc0c8b64a66f99e55531"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.633176 11058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:20.647796 11058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:20.648473 11058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:20.648669 11058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:20.655365 11058 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:43325
I20260619 22:26:20.655377 11124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:43325 every 8 connection(s)
I20260619 22:26:20.656276 11058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:20.659313 11125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:20.659646 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11058
I20260619 22:26:20.659783 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:20.660071 5827 external_mini_cluster.cc:1428] Setting key d5f260e108c635cab4d6c39cc29a16b4
I20260619 22:26:20.666496 11125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3: Bootstrap starting.
I20260619 22:26:20.669243 11125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:20.670105 11125 log.cc:824] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:20.670234 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.661408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.5.176.254:43325)
0619 22:26:20.662039 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.662057 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.662101 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:26:20.662890 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.662914 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.663418 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.663774 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:20.664197 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.665708 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.666568 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.667248 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.667402 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.668187 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:20.668193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:20.668215 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:20.668681 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:20.668749 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:20.668775 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:20.668872 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:20.669010 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 22:26:20.669291 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
I20260619 22:26:20.672211 11125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3: No bootstrap required, opened a new log
I20260619 22:26:20.672940 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:26:20.676167 11125 raft_consensus.cc:348] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d7062c1524a60a1232c1e991115b3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43325 } }
I20260619 22:26:20.676375 11125 raft_consensus.cc:374] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:20.676409 11125 raft_consensus.cc:729] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39d7062c1524a60a1232c1e991115b3, State: Initialized, Role: FOLLOWER
I20260619 22:26:20.676807 11125 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d7062c1524a60a1232c1e991115b3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43325 } }
I20260619 22:26:20.676928 11125 raft_consensus.cc:388] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:20.676973 11125 raft_consensus.cc:482] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:20.677037 11125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:20.677855 11125 raft_consensus.cc:504] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d7062c1524a60a1232c1e991115b3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43325 } }
I20260619 22:26:20.678226 11125 leader_election.cc:302] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39d7062c1524a60a1232c1e991115b3; no voters:
I20260619 22:26:20.678531 11125 leader_election.cc:288] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:20.678654 11130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:20.678905 11130 raft_consensus.cc:686] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [term 1 LEADER]: Becoming Leader. State: Replica: c39d7062c1524a60a1232c1e991115b3, State: Running, Role: LEADER
I20260619 22:26:20.679265 11130 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d7062c1524a60a1232c1e991115b3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43325 } }
I20260619 22:26:20.679831 11125 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:20.681028 11131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39d7062c1524a60a1232c1e991115b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d7062c1524a60a1232c1e991115b3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43325 } } }
I20260619 22:26:20.681028 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39d7062c1524a60a1232c1e991115b3. Latest consensus state: current_term: 1 leader_uuid: "c39d7062c1524a60a1232c1e991115b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d7062c1524a60a1232c1e991115b3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43325 } } }
I20260619 22:26:20.681181 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:20.681181 11131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:20.681579 11140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:20.684866 11140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:20.690233 11140 catalog_manager.cc:1347] Generated new cluster ID: 84dd7b30b9384936adb9e69cc50a0b83
I20260619 22:26:20.690302 11140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:20.699287 11140 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:20.700513 11140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:20.716658 11140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c39d7062c1524a60a1232c1e991115b3: Generated new TSK 0
I20260619 22:26:20.717571 11140 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 22:26:20.742015 11155 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 22:26:20.742121 11155 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 22:26:20.742858 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.726757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.5.176.254:43325)
0619 22:26:20.726924 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.726930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.727079 (+ 149us) server_negotiation.cc:407] Connection header received
0619 22:26:20.727221 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.727224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.727286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.727411 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:26:20.727614 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.728456 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.729298 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.729641 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.729749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.730059 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:26:20.742411 (+ 12352us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 22:26:20.742703 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46780: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 22:26:20.742923 11128 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46780: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 22:26:20.742980 11154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:20.726693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43325 (local address 127.0.0.1:46780)
0619 22:26:20.726989 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:20.727002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:20.727099 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:20.727408 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:20.727410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:20.727416 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:26:20.727510 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:20.727515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.728567 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.728569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:20.729187 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:20.729194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.729764 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.729767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:20.729846 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.729871 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:26:20.742627 (+ 12756us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 22:26:20.742776 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.176.254:43325: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":118,"threads_started":1}
I20260619 22:26:20.743687 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11058
2026-06-19T22:26:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 22:26:20.783161 5827 mini_oidc.cc:152] Starting JWKS server
I20260619 22:26:20.783996 5827 webserver.cc:460] Webserver started at https://127.0.0.1:45659/ using document root <none> and password file <none>
I20260619 22:26:20.784072 5827 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 22:26:20.784266 5827 webserver.cc:460] Webserver started at http://127.0.0.1:40635/ using document root <none> and password file <none>
2026-06-19T22:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:20Z Disabled control of system clock
I20260619 22:26:20.793136 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:40121
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:40121
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45659/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 22:26:20.903227 11161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:20.903509 11161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:20.903548 11161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:20.908102 11161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:20.908172 11161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:20.908192 11161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:20.908210 11161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:20.908226 11161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:26:20.908242 11161 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:26:20.908259 11161 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45659/jwks.json
I20260619 22:26:20.913129 11161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:40121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45659/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:20.914271 11161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:20.915621 11161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:20.924144 11166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:20.924140 11167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:20.924217 11161 server_base.cc:1034] running on GCE node
W20260619 22:26:20.924140 11169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:20.925060 11161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:20.926270 11161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:20.927438 11161 hybrid_clock.cc:648] HybridClock initialized: now 1781907980927407 us; error 43 us; skew 500 ppm
I20260619 22:26:20.929332 11161 webserver.cc:460] Webserver started at http://127.5.176.254:36777/ using document root <none> and password file <none>
I20260619 22:26:20.929910 11161 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:20.929972 11161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:20.930244 11161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:20.932391 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "05827d9f7239450b8398b4d2b9863695"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "15ec209f693bc9eaa7a471b59c3cdae0"
server_key_iv: "e3b1803b50fc3b191a8df9115f86e1bf"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.932998 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "05827d9f7239450b8398b4d2b9863695"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "15ec209f693bc9eaa7a471b59c3cdae0"
server_key_iv: "e3b1803b50fc3b191a8df9115f86e1bf"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.936777 11161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:26:20.939299 11175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:20.940514 11161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:20.940653 11161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "05827d9f7239450b8398b4d2b9863695"
format_stamp: "Formatted at 2026-06-19 22:26:20 on dist-test-slave-bt27"
server_key: "15ec209f693bc9eaa7a471b59c3cdae0"
server_key_iv: "e3b1803b50fc3b191a8df9115f86e1bf"
server_key_version: "encryptionkey@0"
I20260619 22:26:20.940768 11161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:20.957520 11161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:20.958319 11161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:20.958526 11161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:20.965889 11161 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:40121
I20260619 22:26:20.965917 11227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:40121 every 8 connection(s)
I20260619 22:26:20.966920 11161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:20.969144 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11161
I20260619 22:26:20.969269 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:20.969540 5827 external_mini_cluster.cc:1428] Setting key 3fc60ab54311e3c08d8e5b9fb616f0ca
I20260619 22:26:20.970538 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:20.977450 11228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695: Bootstrap starting.
I20260619 22:26:20.980319 11228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:20.980367 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:20.971005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.5.176.254:40121)
0619 22:26:20.971668 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:20.971688 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:20.971741 (+ 53us) server_negotiation.cc:407] Connection header received
0619 22:26:20.972614 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:20.972636 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:20.973149 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:20.973514 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:20.973970 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.975426 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.976447 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:20.977123 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:20.977282 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:20.978234 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:20.978241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:20.978257 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:20.978782 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:20.978860 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:20.978889 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:20.978997 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:20.979150 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 22:26:20.979433 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260619 22:26:20.981076 11228 log.cc:824] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:20.983335 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:26:20.983404 11228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695: No bootstrap required, opened a new log
I20260619 22:26:20.987033 11228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05827d9f7239450b8398b4d2b9863695" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40121 } }
I20260619 22:26:20.987259 11228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:20.987301 11228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05827d9f7239450b8398b4d2b9863695, State: Initialized, Role: FOLLOWER
I20260619 22:26:20.987793 11228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05827d9f7239450b8398b4d2b9863695" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40121 } }
I20260619 22:26:20.987977 11228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:20.988045 11228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:20.988139 11228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:20.989048 11228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05827d9f7239450b8398b4d2b9863695" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40121 } }
I20260619 22:26:20.989383 11228 leader_election.cc:302] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05827d9f7239450b8398b4d2b9863695; no voters:
I20260619 22:26:20.989679 11228 leader_election.cc:288] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:20.989845 11233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:20.990089 11233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [term 1 LEADER]: Becoming Leader. State: Replica: 05827d9f7239450b8398b4d2b9863695, State: Running, Role: LEADER
I20260619 22:26:20.990399 11233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05827d9f7239450b8398b4d2b9863695" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40121 } }
I20260619 22:26:20.990872 11228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:20.992000 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05827d9f7239450b8398b4d2b9863695. Latest consensus state: current_term: 1 leader_uuid: "05827d9f7239450b8398b4d2b9863695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05827d9f7239450b8398b4d2b9863695" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40121 } } }
I20260619 22:26:20.992138 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:20.992316 11234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05827d9f7239450b8398b4d2b9863695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05827d9f7239450b8398b4d2b9863695" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40121 } } }
I20260619 22:26:20.992468 11234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:20.992549 11242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:20.995585 11242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:21.001427 11242 catalog_manager.cc:1347] Generated new cluster ID: 8653a047fcbf4381b5164f43f95152e1
I20260619 22:26:21.001511 11242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:21.015548 11242 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:21.016930 11242 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:21.029948 11242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05827d9f7239450b8398b4d2b9863695: Generated new TSK 0
I20260619 22:26:21.030726 11242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:21.048024 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:21.042776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.5.176.254:40121)
0619 22:26:21.042919 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:21.042924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:21.043048 (+ 124us) server_negotiation.cc:407] Connection header received
0619 22:26:21.043202 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:21.043205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:21.043251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:21.043338 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:26:21.043587 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.044208 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.045491 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.045738 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.045847 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:21.046246 (+ 399us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:26:21.047653 (+ 1407us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 22:26:21.047883 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 22:26:21.048094 11231 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 22:26:21.049808 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:21.042645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40121 (local address 127.0.0.1:33708)
0619 22:26:21.042958 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:21.042972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:21.043085 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:21.043362 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:21.043365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:21.043373 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:26:21.043489 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:21.043495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.044352 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.044356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:21.045367 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:21.045377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.045872 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.045876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:21.045963 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:21.045990 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:26:21.047829 (+ 1839us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 22:26:21.049630 (+ 1801us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.176.254:40121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1603,"thread_start_us":108,"threads_started":1}
I20260619 22:26:21.050405 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11161
E20260619 22:26:21.058799 11258 webserver.cc:533] Webserver: sslize failed:
E20260619 22:26:21.058878 11258 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T22:26:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T22:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:21Z Disabled control of system clock
I20260619 22:26:21.098719 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:46247
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:46247
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/testchainca.cert with env {}
W20260619 22:26:21.211673 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:21.212015 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:21.212095 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:21.216231 11262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:21.216336 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:21.216365 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:21.216401 11262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:21.216460 11262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 22:26:21.216499 11262 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 22:26:21.216547 11262 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 22:26:21.222159 11262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:46247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:46247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:21.223286 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:21.224382 11262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:21.233174 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:21.233256 11262 server_base.cc:1034] running on GCE node
W20260619 22:26:21.233214 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:21.233214 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:21.234257 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:21.235365 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:21.236548 11262 hybrid_clock.cc:648] HybridClock initialized: now 1781907981236529 us; error 32 us; skew 500 ppm
I20260619 22:26:21.238642 11262 webserver.cc:460] Webserver started at http://127.5.176.254:46841/ using document root <none> and password file <none>
I20260619 22:26:21.239202 11262 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:21.239261 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:21.239424 11262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:21.241464 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "bc8d08498fb34890927bda7a6eac26bb"
format_stamp: "Formatted at 2026-06-19 22:26:21 on dist-test-slave-bt27"
server_key: "88de3e378c32f66cc5b327e1f0054c84"
server_key_iv: "55a7b3da481e4fcba945daf7eba80286"
server_key_version: "encryptionkey@0"
I20260619 22:26:21.242004 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "bc8d08498fb34890927bda7a6eac26bb"
format_stamp: "Formatted at 2026-06-19 22:26:21 on dist-test-slave-bt27"
server_key: "88de3e378c32f66cc5b327e1f0054c84"
server_key_iv: "55a7b3da481e4fcba945daf7eba80286"
server_key_version: "encryptionkey@0"
I20260619 22:26:21.245872 11262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:26:21.248582 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:21.250263 11262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:21.250403 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "bc8d08498fb34890927bda7a6eac26bb"
format_stamp: "Formatted at 2026-06-19 22:26:21 on dist-test-slave-bt27"
server_key: "88de3e378c32f66cc5b327e1f0054c84"
server_key_iv: "55a7b3da481e4fcba945daf7eba80286"
server_key_version: "encryptionkey@0"
I20260619 22:26:21.250555 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:21.264925 11262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:21.265632 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:21.265851 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:21.273007 11262 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:46247
I20260619 22:26:21.273031 11328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:46247 every 8 connection(s)
I20260619 22:26:21.273977 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:21.274700 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11262
I20260619 22:26:21.274857 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:21.275195 5827 external_mini_cluster.cc:1428] Setting key a2f4141da618dc46ef990dcbda2f66ae
I20260619 22:26:21.277333 11329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:21.283988 11329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb: Bootstrap starting.
I20260619 22:26:21.285337 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:21.276482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.5.176.254:46247)
0619 22:26:21.277118 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:21.277134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:21.277768 (+ 634us) server_negotiation.cc:407] Connection header received
0619 22:26:21.278583 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:21.278600 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:21.279031 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:21.279327 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:21.279674 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.280642 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.281866 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.282435 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.282567 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:21.283463 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:21.283468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:21.283481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:21.283908 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:21.283966 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:21.283986 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:21.284064 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:21.284294 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 22:26:21.284608 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":148,"threads_started":1}
I20260619 22:26:21.286525 11329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:21.287389 11329 log.cc:824] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:21.288491 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:26:21.289795 11329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb: No bootstrap required, opened a new log
I20260619 22:26:21.293370 11329 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d08498fb34890927bda7a6eac26bb" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46247 } }
I20260619 22:26:21.293608 11329 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:21.293643 11329 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8d08498fb34890927bda7a6eac26bb, State: Initialized, Role: FOLLOWER
I20260619 22:26:21.294003 11329 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d08498fb34890927bda7a6eac26bb" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46247 } }
I20260619 22:26:21.294111 11329 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:21.294214 11329 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:21.294313 11329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:21.295303 11329 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d08498fb34890927bda7a6eac26bb" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46247 } }
I20260619 22:26:21.295671 11329 leader_election.cc:302] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc8d08498fb34890927bda7a6eac26bb; no voters:
I20260619 22:26:21.296068 11329 leader_election.cc:288] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:21.296195 11334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:21.296439 11334 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [term 1 LEADER]: Becoming Leader. State: Replica: bc8d08498fb34890927bda7a6eac26bb, State: Running, Role: LEADER
I20260619 22:26:21.296765 11334 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d08498fb34890927bda7a6eac26bb" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46247 } }
I20260619 22:26:21.297405 11329 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:21.298477 11335 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc8d08498fb34890927bda7a6eac26bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d08498fb34890927bda7a6eac26bb" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46247 } } }
I20260619 22:26:21.298617 11335 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:21.298897 11336 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc8d08498fb34890927bda7a6eac26bb. Latest consensus state: current_term: 1 leader_uuid: "bc8d08498fb34890927bda7a6eac26bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d08498fb34890927bda7a6eac26bb" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46247 } } }
I20260619 22:26:21.299000 11336 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:21.299065 11343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:21.302232 11343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:21.307904 11343 catalog_manager.cc:1347] Generated new cluster ID: d1688985a10c44d1a9a273f73dc55e4d
I20260619 22:26:21.307979 11343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:21.326929 11343 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:21.328205 11343 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:21.348821 11343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc8d08498fb34890927bda7a6eac26bb: Generated new TSK 0
I20260619 22:26:21.349753 11343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:21.418638 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:21.358936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.5.176.254:46247)
0619 22:26:21.359079 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:21.359084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:21.359231 (+ 147us) server_negotiation.cc:407] Connection header received
0619 22:26:21.359374 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:21.359377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:21.359437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:21.359559 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0619 22:26:21.359762 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.360615 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.361520 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.361875 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.361976 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:21.362369 (+ 393us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 22:26:21.418072 (+ 55703us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 22:26:21.418475 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35652: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 22:26:21.418732 11332 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35652: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 22:26:21.418731 11358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:21.358859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46247 (local address 127.0.0.1:35652)
0619 22:26:21.359146 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:21.359159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:21.359257 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:21.359558 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:21.359561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:21.359567 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 22:26:21.359663 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:21.359668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.360738 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.360741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:21.361404 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:21.361413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:21.361996 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:21.361999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:21.362075 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:21.362100 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 22:26:21.418363 (+ 56263us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 22:26:21.418551 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.176.254:46247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260619 22:26:21.419806 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11262
2026-06-19T22:26:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (349 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:21 dist-test-slave-bt27 krb5kdc[11370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:21 dist-test-slave-bt27 krb5kdc[11370](info): set up 2 sockets
Jun 19 22:26:21 dist-test-slave-bt27 krb5kdc[11370](info): commencing operation
krb5kdc: starting...
W20260619 22:26:22.719076 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:22 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:22.842864 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34873
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:34873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:22.953740 11383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:22.954090 11383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:22.954159 11383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:22.958375 11383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:22.958463 11383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:22.958499 11383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:22.958539 11383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:22.958575 11383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:22.964232 11383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:34873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:22.965411 11383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:22.966537 11383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:22.975000 11389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:22.974987 11391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:22.974987 11388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:22.975134 11383 server_base.cc:1034] running on GCE node
I20260619 22:26:22.975876 11383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:22.976975 11383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:22.978192 11383 hybrid_clock.cc:648] HybridClock initialized: now 1781907982978120 us; error 32 us; skew 500 ppm
Jun 19 22:26:22 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907982, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:22.981101 11383 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:22.981968 11383 webserver.cc:460] Webserver started at http://127.5.176.254:43391/ using document root <none> and password file <none>
I20260619 22:26:22.982596 11383 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:22.982662 11383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:22.982892 11383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:22.984918 11383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "7ce1414bf6624f7b86940b990cf9da8c"
format_stamp: "Formatted at 2026-06-19 22:26:22 on dist-test-slave-bt27"
server_key: "c4ac33976513b9c2212834dfa4ef9490"
server_key_iv: "e406c14d2e2af78952f26997c6132e7e"
server_key_version: "encryptionkey@0"
I20260619 22:26:22.985466 11383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "7ce1414bf6624f7b86940b990cf9da8c"
format_stamp: "Formatted at 2026-06-19 22:26:22 on dist-test-slave-bt27"
server_key: "c4ac33976513b9c2212834dfa4ef9490"
server_key_iv: "e406c14d2e2af78952f26997c6132e7e"
server_key_version: "encryptionkey@0"
I20260619 22:26:22.989180 11383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 22:26:22.991643 11398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:22.993031 11383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:22.993187 11383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "7ce1414bf6624f7b86940b990cf9da8c"
format_stamp: "Formatted at 2026-06-19 22:26:22 on dist-test-slave-bt27"
server_key: "c4ac33976513b9c2212834dfa4ef9490"
server_key_iv: "e406c14d2e2af78952f26997c6132e7e"
server_key_version: "encryptionkey@0"
I20260619 22:26:22.993319 11383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:23.011767 11383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:23.015064 11383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:23.015287 11383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:23.022125 11383 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:34873
I20260619 22:26:23.022217 11450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:34873 every 8 connection(s)
I20260619 22:26:23.023118 11383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:23.025794 11451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:23.029579 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11383
I20260619 22:26:23.029727 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:23.030002 5827 external_mini_cluster.cc:1428] Setting key ee8619bd4f3993e80b021ef58ec5beba
I20260619 22:26:23.031725 11451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c: Bootstrap starting.
I20260619 22:26:23.034308 11451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:23.035045 11451 log.cc:824] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:23.037222 11451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c: No bootstrap required, opened a new log
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:23.041862 11451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce1414bf6624f7b86940b990cf9da8c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34873 } }
I20260619 22:26:23.042186 11451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:23.042258 11451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce1414bf6624f7b86940b990cf9da8c, State: Initialized, Role: FOLLOWER
I20260619 22:26:23.042758 11451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce1414bf6624f7b86940b990cf9da8c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34873 } }
I20260619 22:26:23.042928 11451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:23.043021 11451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:23.043119 11451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:23.043998 11451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce1414bf6624f7b86940b990cf9da8c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34873 } }
I20260619 22:26:23.044350 11451 leader_election.cc:302] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ce1414bf6624f7b86940b990cf9da8c; no voters:
I20260619 22:26:23.044700 11451 leader_election.cc:288] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:23.044808 11456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:23.045029 11456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [term 1 LEADER]: Becoming Leader. State: Replica: 7ce1414bf6624f7b86940b990cf9da8c, State: Running, Role: LEADER
I20260619 22:26:23.045433 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.031527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.5.176.254:34873)
0619 22:26:23.031946 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.031965 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.031999 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:26:23.032718 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.032735 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.033056 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.033356 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.034269 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.035395 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.036356 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.037021 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.037183 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.039984 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.039993 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.040014 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.040054 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.042050 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.042614 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.042619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.042625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.042760 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.043116 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.043119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.043121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.043565 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.043789 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.044067 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.044393 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":147,"threads_started":1}
I20260619 22:26:23.045383 11456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce1414bf6624f7b86940b990cf9da8c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34873 } }
I20260619 22:26:23.046057 11451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:23.046619 11456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce1414bf6624f7b86940b990cf9da8c. Latest consensus state: current_term: 1 leader_uuid: "7ce1414bf6624f7b86940b990cf9da8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce1414bf6624f7b86940b990cf9da8c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34873 } } }
I20260619 22:26:23.046736 11456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:23.047413 11458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ce1414bf6624f7b86940b990cf9da8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce1414bf6624f7b86940b990cf9da8c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34873 } } }
I20260619 22:26:23.047514 11458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:23.047883 11461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:23.051024 11461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:23.056811 11461 catalog_manager.cc:1347] Generated new cluster ID: 8f9cd12b18b14d2cb5cb6d50db17171e
I20260619 22:26:23.056910 11461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:23.065251 11461 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:23.066488 11461 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:23.081014 11461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ce1414bf6624f7b86940b990cf9da8c: Generated new TSK 0
I20260619 22:26:23.081836 11461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:23.117835 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34873
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:23.233968 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:23.234292 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:23.234344 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:23.238457 11477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:23.238519 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:23.238579 11477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:23.243981 11477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:23.245060 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:23.246045 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:23.256611 11485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:23.256842 11483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:23.256652 11482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:23.256983 11477 server_base.cc:1034] running on GCE node
I20260619 22:26:23.257481 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:23.258442 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:23.259693 11477 hybrid_clock.cc:648] HybridClock initialized: now 1781907983259668 us; error 38 us; skew 500 ppm
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.262660 11477 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:23.263595 11477 webserver.cc:460] Webserver started at http://127.5.176.193:41255/ using document root <none> and password file <none>
I20260619 22:26:23.264176 11477 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:23.264266 11477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:23.264530 11477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:23.266670 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "b07e0ad707cb4993b5e6b37f9cd07b27"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "3e79f3de8ce62bf2c3ade4b5fdf11a89"
server_key_iv: "b1afa3be8c9d120144328128e3516779"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.267242 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "b07e0ad707cb4993b5e6b37f9cd07b27"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "3e79f3de8ce62bf2c3ade4b5fdf11a89"
server_key_iv: "b1afa3be8c9d120144328128e3516779"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.270998 11477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:23.273687 11492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.275168 11477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:23.275338 11477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "b07e0ad707cb4993b5e6b37f9cd07b27"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "3e79f3de8ce62bf2c3ade4b5fdf11a89"
server_key_iv: "b1afa3be8c9d120144328128e3516779"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.275472 11477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:23.298558 11477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:23.302050 11477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:23.302294 11477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:23.302922 11477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:23.303768 11477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:23.303840 11477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.303941 11477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:23.303990 11477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.312932 11477 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:35813
I20260619 22:26:23.312952 11605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:35813 every 8 connection(s)
I20260619 22:26:23.313920 11477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:23.314790 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11477
I20260619 22:26:23.314932 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:23.315234 5827 external_mini_cluster.cc:1428] Setting key 1453d9f4a6cc01d8e987ce9fd7db30a3
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:23.328979 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.315978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:43495 (local address 127.5.176.254:34873)
0619 22:26:23.316119 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.316122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.317160 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 22:26:23.318809 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.318812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.318856 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.318927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.320252 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.320793 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.321967 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.322259 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.322369 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.325103 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.325106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.325108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.325130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.326662 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.327238 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.327243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.327245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.327328 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.327741 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.327746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.327748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.327977 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.328097 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.328642 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.328780 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:26:23.329866 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.316456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34873 (local address 127.5.176.193:43495)
0619 22:26:23.316989 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:23.317042 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:23.318556 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:23.319117 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:23.319130 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:23.319600 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:23.320093 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:23.320107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.320931 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.320934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:23.321839 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:23.321849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.322420 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.322424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:23.322583 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.323227 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:23.323271 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:23.324934 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:23.326850 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:23.326855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:23.326871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:23.327134 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:23.327472 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:23.327475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:23.327477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:23.327642 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:23.328096 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:23.328103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:23.328408 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 22:26:23.328604 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 22:26:23.328934 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20260619 22:26:23.331269 11606 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34873
I20260619 22:26:23.331554 11606 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:23.332100 11606 heartbeater.cc:511] Master 127.5.176.254:34873 requested a full tablet report, sending...
I20260619 22:26:23.333912 11415 ts_manager.cc:194] Registered new tserver with Master: b07e0ad707cb4993b5e6b37f9cd07b27 (127.5.176.193:35813)
I20260619 22:26:23.335186 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:43495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:23.345693 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34873
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:23.457275 11611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:23.457551 11611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:23.457597 11611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:23.461916 11611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:23.461983 11611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:23.462044 11611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:23.467592 11611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:23.468755 11611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:23.469784 11611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:23.479722 11617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:23.479703 11619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:23.479755 11611 server_base.cc:1034] running on GCE node
W20260619 22:26:23.479703 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:23.480451 11611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:23.481165 11611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:23.482333 11611 hybrid_clock.cc:648] HybridClock initialized: now 1781907983482283 us; error 63 us; skew 500 ppm
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.485183 11611 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:23.486222 11611 webserver.cc:460] Webserver started at http://127.5.176.194:46767/ using document root <none> and password file <none>
I20260619 22:26:23.486827 11611 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:23.486891 11611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:23.487120 11611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:23.489246 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "0e6b1edb5cf94ff38ec0ead3e49000c7"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "2cab1747664d3d03ef1c74b3025b1916"
server_key_iv: "5b24c41e0c836d9a07c5bfd4c44dc9aa"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.489835 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "0e6b1edb5cf94ff38ec0ead3e49000c7"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "2cab1747664d3d03ef1c74b3025b1916"
server_key_iv: "5b24c41e0c836d9a07c5bfd4c44dc9aa"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.493760 11611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 22:26:23.496466 11626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.497818 11611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:23.497916 11611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "0e6b1edb5cf94ff38ec0ead3e49000c7"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "2cab1747664d3d03ef1c74b3025b1916"
server_key_iv: "5b24c41e0c836d9a07c5bfd4c44dc9aa"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.498008 11611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:23.509182 11611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:23.512298 11611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:23.512452 11611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:23.512970 11611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:23.513751 11611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:23.513797 11611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.513837 11611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:23.513864 11611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.523083 11611 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:45099
I20260619 22:26:23.523100 11739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:45099 every 8 connection(s)
I20260619 22:26:23.524026 11611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:23.532629 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11611
I20260619 22:26:23.532791 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:23.533116 5827 external_mini_cluster.cc:1428] Setting key 06813d6d4c671729c5365e992871333c
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:23.539484 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.525957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:44297 (local address 127.5.176.254:34873)
0619 22:26:23.526120 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.526125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.527171 (+ 1046us) server_negotiation.cc:407] Connection header received
0619 22:26:23.528545 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.528548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.528600 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.528683 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.530111 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.530739 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.531930 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.532149 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.532241 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.535205 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.535209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.535211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.535243 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.537070 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.537629 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.537635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.537640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.537731 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.538124 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.538130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.538152 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.538415 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.538522 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.539093 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.539243 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:26:23.540369 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.526439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34873 (local address 127.5.176.194:44297)
0619 22:26:23.526995 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:23.527043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:23.528314 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:23.528897 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:23.528904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:23.529288 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:23.529935 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:23.529950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.530876 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.530879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:23.531806 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:23.531816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.532359 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.532364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:23.532585 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.533243 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:23.533265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:23.535029 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:23.537204 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:23.537209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:23.537223 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:23.537518 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:23.537842 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:23.537845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:23.537847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:23.538016 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:23.538526 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:23.538532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:23.538825 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 22:26:23.539008 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 22:26:23.539386 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":123,"threads_started":1}
I20260619 22:26:23.541687 11740 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34873
I20260619 22:26:23.541965 11740 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:23.542623 11740 heartbeater.cc:511] Master 127.5.176.254:34873 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:26:23.543892 11415 ts_manager.cc:194] Registered new tserver with Master: 0e6b1edb5cf94ff38ec0ead3e49000c7 (127.5.176.194:45099)
I20260619 22:26:23.544560 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:44297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:23.561264 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34873
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:23.704036 11745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:23.704522 11745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:23.704632 11745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:23.714119 11745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:23.714275 11745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:23.714447 11745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:23.726656 11745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:23.728276 11745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:23.729771 11745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:23.741974 11753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:23.742036 11750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:23.742036 11751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:23.742365 11745 server_base.cc:1034] running on GCE node
I20260619 22:26:23.742993 11745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:23.743866 11745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:23.745078 11745 hybrid_clock.cc:648] HybridClock initialized: now 1781907983745042 us; error 45 us; skew 500 ppm
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.748935 11745 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:23.749936 11745 webserver.cc:460] Webserver started at http://127.5.176.195:45523/ using document root <none> and password file <none>
I20260619 22:26:23.750555 11745 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:23.750617 11745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:23.750831 11745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:23.752909 11745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "546d72fad74b44c3b8d1906507d7a733"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "395f70033bddd550d5c1854014542c90"
server_key_iv: "1f5c6223ab91668026e269d6221ca68a"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.753487 11745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "546d72fad74b44c3b8d1906507d7a733"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "395f70033bddd550d5c1854014542c90"
server_key_iv: "1f5c6223ab91668026e269d6221ca68a"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.757452 11745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:26:23.760190 11760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.761603 11745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:23.761742 11745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "546d72fad74b44c3b8d1906507d7a733"
format_stamp: "Formatted at 2026-06-19 22:26:23 on dist-test-slave-bt27"
server_key: "395f70033bddd550d5c1854014542c90"
server_key_iv: "1f5c6223ab91668026e269d6221ca68a"
server_key_version: "encryptionkey@0"
I20260619 22:26:23.761860 11745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:23.779170 11745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:23.782744 11745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:23.782963 11745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:23.783628 11745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:23.784492 11745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:23.784564 11745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.784646 11745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:23.784694 11745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:23.795440 11745 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42083
I20260619 22:26:23.795511 11873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42083 every 8 connection(s)
I20260619 22:26:23.796473 11745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:23.798569 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11745
I20260619 22:26:23.798719 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:23.799134 5827 external_mini_cluster.cc:1428] Setting key 13755a2911f7ff7affebaf6a3e7e06ba
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:23.811694 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.798492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:52087 (local address 127.5.176.254:34873)
0619 22:26:23.798631 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.798636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.799667 (+ 1031us) server_negotiation.cc:407] Connection header received
0619 22:26:23.801370 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.801373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.801427 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.801502 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.802863 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.803401 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.804533 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.804753 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.804864 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.807377 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.807384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.807389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.807429 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.809264 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.809794 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.809802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.809804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.809917 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.810328 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.810334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.810338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.810625 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.810748 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.811275 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.811435 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:23.812352 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.798985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34873 (local address 127.5.176.195:52087)
0619 22:26:23.799511 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:23.799565 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:23.801117 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:23.801691 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:23.801697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:23.802039 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:23.802732 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:23.802746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.803521 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.803525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:23.804389 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:23.804407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.804897 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.804901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:23.805082 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.805741 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:23.805756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:23.807198 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:23.809414 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:23.809418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:23.809430 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:23.809680 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:23.810027 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:23.810029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:23.810031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:23.810234 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:23.810741 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:23.810766 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:23.810999 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 22:26:23.811161 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 22:26:23.811412 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20260619 22:26:23.813416 11874 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34873
I20260619 22:26:23.813668 11874 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:23.814095 11874 heartbeater.cc:511] Master 127.5.176.254:34873 requested a full tablet report, sending...
I20260619 22:26:23.815178 11415 ts_manager.cc:194] Registered new tserver with Master: 546d72fad74b44c3b8d1906507d7a733 (127.5.176.195:42083)
I20260619 22:26:23.816112 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:52087
I20260619 22:26:23.826122 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/test-admin.keytab.
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.843451 5827 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 22:26:23.846693 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:23.845032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.5.176.193:35813)
0619 22:26:23.845319 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.845333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.845357 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:26:23.845426 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.845431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.845618 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.845734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.846460 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36192: BlockingRecv error: recv got EOF from 127.0.0.1:36192 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
W20260619 22:26:23.847114 5827 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc' of type FILE
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.848693 5827 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.849872 5827 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:26:23.861219 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.850873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.5.176.193:35813)
0619 22:26:23.850990 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.850994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.851184 (+ 190us) server_negotiation.cc:407] Connection header received
0619 22:26:23.851337 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.851340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.851391 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.851474 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.852055 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.852814 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.853761 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.854221 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.854344 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.856620 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.856635 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.856657 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.856699 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.859047 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.859541 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.859551 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.859554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.859682 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.860011 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.860016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.860019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.860431 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.860669 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.860856 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.861026 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 22:26:23.865022 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:23.864136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.5.176.193:35813)
0619 22:26:23.864273 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.864276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.864321 (+ 45us) server_negotiation.cc:407] Connection header received
0619 22:26:23.864453 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.864456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.864490 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.864553 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.864925 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36208: BlockingRecv error: recv got EOF from 127.0.0.1:36208 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 22:26:23.865132 5827 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc' of type FILE
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.866334 5827 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.867239 5827 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:26:23.875304 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.868013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.5.176.193:35813)
0619 22:26:23.868119 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.868122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.868148 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:26:23.868247 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.868249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.868283 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.868350 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.868823 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.869314 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.870057 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.870348 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.870432 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.872609 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.872612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.872614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.872636 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.873905 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.874370 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.874372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.874373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.874444 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.874665 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.874667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.874668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.874842 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.874956 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.875108 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.875193 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 22:26:23.877739 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:23.876989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.5.176.193:35813)
0619 22:26:23.877117 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.877120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.877165 (+ 45us) server_negotiation.cc:407] Connection header received
0619 22:26:23.877253 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.877255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.877289 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.877348 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.877655 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36228: BlockingRecv error: recv got EOF from 127.0.0.1:36228 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 22:26:23.878064 5827 init.cc:244] Error while opening credential cache '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781907951519384-5827-0/krb5kdc/krb5cc)
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.879086 5827 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:23.879993 5827 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:26:23.888337 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:23.880883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.5.176.193:35813)
0619 22:26:23.881008 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:23.881012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:23.881100 (+ 88us) server_negotiation.cc:407] Connection header received
0619 22:26:23.881237 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:23.881241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:23.881280 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:23.881364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:23.881845 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.882401 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.883171 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:23.883389 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:23.883491 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:23.885501 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:23.885505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:23.885507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:23.885537 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:23.886727 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.887205 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.887210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.887211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.887288 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:23.887553 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:23.887558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:23.887560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:23.887715 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:23.887880 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:23.888087 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 22:26:23.888184 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:26:23.889317 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11477
I20260619 22:26:23.895182 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11611
I20260619 22:26:23.900837 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11745
I20260619 22:26:23.907779 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11383
2026-06-19T22:26:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2234 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11894](info): set up 2 sockets
Jun 19 22:26:23 dist-test-slave-bt27 krb5kdc[11894](info): commencing operation
krb5kdc: starting...
W20260619 22:26:24.942553 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab.
Entry for principal oryx/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab.
I20260619 22:26:25.069447 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:41209
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab
--principal=oryx/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:41209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:25.182898 11907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:25.183188 11907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:25.183230 11907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:25.187247 11907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:25.187312 11907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:25.187332 11907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:25.187348 11907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:25.187364 11907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:25.193061 11907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:41209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab
--principal=oryx/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:41209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:25.194257 11907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:25.195288 11907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:25.204408 11913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:25.204408 11915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:25.204448 11912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.204644 11907 server_base.cc:1034] running on GCE node
I20260619 22:26:25.205453 11907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:25.206698 11907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:25.207918 11907 hybrid_clock.cc:648] HybridClock initialized: now 1781907985207884 us; error 51 us; skew 500 ppm
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:25.214056 11907 init.cc:377] Logged in from keytab as oryx/127.5.176.254@KRBTEST.COM (short username oryx)
I20260619 22:26:25.215173 11907 webserver.cc:460] Webserver started at http://127.5.176.254:40317/ using document root <none> and password file <none>
I20260619 22:26:25.215777 11907 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:25.215870 11907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:25.216112 11907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:25.218353 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "f5262501ae174235a345e305a6749db8"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "44227109630d72ab4c50d313ce420145"
server_key_iv: "0b8eaed337bca5a2620dacc55c55bfa6"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.218958 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "f5262501ae174235a345e305a6749db8"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "44227109630d72ab4c50d313ce420145"
server_key_iv: "0b8eaed337bca5a2620dacc55c55bfa6"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.223040 11907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:25.225713 11922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.227231 11907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:25.227420 11907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "f5262501ae174235a345e305a6749db8"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "44227109630d72ab4c50d313ce420145"
server_key_iv: "0b8eaed337bca5a2620dacc55c55bfa6"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.227587 11907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:25.239907 11907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:25.243227 11907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:25.243459 11907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:25.250697 11907 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:41209
I20260619 22:26:25.251001 11974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:41209 every 8 connection(s)
I20260619 22:26:25.251719 11907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:25.254539 11975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:25.256172 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 11907
I20260619 22:26:25.256342 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:25.256625 5827 external_mini_cluster.cc:1428] Setting key 6e085b2349275881667af939e4682b6f
I20260619 22:26:25.261421 11975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Bootstrap starting.
I20260619 22:26:25.264052 11975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:25.265391 11975 log.cc:824] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:25.267961 11975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: No bootstrap required, opened a new log
I20260619 22:26:25.271361 11975 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } }
I20260619 22:26:25.271608 11975 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:25.271658 11975 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5262501ae174235a345e305a6749db8, State: Initialized, Role: FOLLOWER
I20260619 22:26:25.272114 11975 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } }
I20260619 22:26:25.272269 11975 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:25.272352 11975 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:25.272468 11975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:25.272681 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.258047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.5.176.254:41209)
0619 22:26:25.258538 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:25.258559 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:25.258603 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:26:25.259491 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:25.259515 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:25.259953 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:25.260335 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:25.261061 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.262510 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.263420 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.264105 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.264270 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:25.267854 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:25.267867 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:25.267881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:25.267908 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:25.269628 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:25.270081 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:25.270089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:25.270100 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:25.270258 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:25.270543 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:25.270547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:25.270549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:25.270942 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:25.271146 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:25.271439 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 22:26:25.271737 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":105,"threads_started":1}
I20260619 22:26:25.273649 11975 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } }
I20260619 22:26:25.273993 11975 leader_election.cc:302] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5262501ae174235a345e305a6749db8; no voters:
I20260619 22:26:25.274422 11975 leader_election.cc:288] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:25.274474 11980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:25.274655 11980 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 LEADER]: Becoming Leader. State: Replica: f5262501ae174235a345e305a6749db8, State: Running, Role: LEADER
I20260619 22:26:25.275228 11980 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } }
I20260619 22:26:25.275647 11975 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:25.276939 11982 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } }
I20260619 22:26:25.277052 11982 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:25.277437 11981 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5262501ae174235a345e305a6749db8. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } }
I20260619 22:26:25.277545 11981 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:25.277561 11990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:25.280840 11990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:25.286551 11990 catalog_manager.cc:1347] Generated new cluster ID: f1068e962853462ba604b42122c242c4
I20260619 22:26:25.286638 11990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:25.295706 11990 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:25.297091 11990 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:25.312119 11990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Generated new TSK 0
I20260619 22:26:25.313027 11990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab.
Entry for principal oryx/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab.
I20260619 22:26:25.351163 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41209
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:25.465063 12001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:25.465330 12001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:25.465373 12001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:25.469488 12001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:25.469554 12001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:25.469612 12001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:25.474932 12001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:25.476135 12001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:25.477332 12001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:25.487469 12009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:25.487468 12007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.487542 12001 server_base.cc:1034] running on GCE node
W20260619 22:26:25.487468 12006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.488093 12001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:25.488709 12001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:25.489890 12001 hybrid_clock.cc:648] HybridClock initialized: now 1781907985489852 us; error 48 us; skew 500 ppm
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:25.493023 12001 init.cc:377] Logged in from keytab as oryx/127.5.176.193@KRBTEST.COM (short username oryx)
I20260619 22:26:25.494014 12001 webserver.cc:460] Webserver started at http://127.5.176.193:38233/ using document root <none> and password file <none>
I20260619 22:26:25.494652 12001 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:25.494719 12001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:25.494976 12001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:25.497115 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "60620add323f9707aa70b9fff3ff45d4"
server_key_iv: "ae763f4b455f3714660627e7b6154d36"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.497709 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "60620add323f9707aa70b9fff3ff45d4"
server_key_iv: "ae763f4b455f3714660627e7b6154d36"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.501685 12001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:26:25.504489 12016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.505957 12001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:25.506116 12001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "60620add323f9707aa70b9fff3ff45d4"
server_key_iv: "ae763f4b455f3714660627e7b6154d36"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.506313 12001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:25.528687 12001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:25.532075 12001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:25.532231 12001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:25.532764 12001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:25.533529 12001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:25.533574 12001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.533613 12001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:25.533632 12001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.543056 12001 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:39413
I20260619 22:26:25.543082 12129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:39413 every 8 connection(s)
I20260619 22:26:25.543982 12001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:25.547915 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12001
I20260619 22:26:25.548079 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:25.548441 5827 external_mini_cluster.cc:1428] Setting key 4a4820f71815bd2d805a93d5d9d56ffe
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:26:25.558482 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.545955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:56607 (local address 127.5.176.254:41209)
0619 22:26:25.546087 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:25.546099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:25.547028 (+ 929us) server_negotiation.cc:407] Connection header received
0619 22:26:25.548341 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:25.548346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:25.548414 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:25.548535 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:25.549756 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.550354 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.551620 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.551870 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.551969 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:25.554622 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:25.554625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:25.554627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:25.554651 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:25.556238 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:25.556814 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:25.556819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:25.556821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:25.556906 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:25.557263 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:25.557266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:25.557268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:25.557455 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:25.557563 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:25.558165 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 22:26:25.558308 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:25.559440 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.546370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:56607)
0619 22:26:25.546852 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:25.546894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:25.548083 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:25.548729 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:25.548736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:25.549125 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:25.549604 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:25.549617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.550478 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.550482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:25.551477 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:25.551491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.552037 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.552043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:25.552269 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:25.552959 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:25.552975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:25.554443 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:25.556395 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:25.556402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:25.556418 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:25.556693 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:25.557001 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:25.557006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:25.557008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:25.557155 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:25.557562 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:25.557569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:25.557898 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 22:26:25.558098 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 22:26:25.558505 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:25.560958 12130 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41209
I20260619 22:26:25.561232 12130 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:25.561867 12130 heartbeater.cc:511] Master 127.5.176.254:41209 requested a full tablet report, sending...
I20260619 22:26:25.563812 11939 ts_manager.cc:194] Registered new tserver with Master: f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413)
I20260619 22:26:25.564981 11939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:56607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab.
Entry for principal oryx/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab.
I20260619 22:26:25.577584 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41209
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:25.694342 12135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:25.694617 12135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:25.694661 12135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:25.698740 12135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:25.698801 12135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:25.698904 12135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:25.704330 12135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:25.705430 12135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:25.706482 12135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:25.716447 12141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:25.716465 12140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.716595 12135 server_base.cc:1034] running on GCE node
W20260619 22:26:25.716461 12143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.717094 12135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:25.717670 12135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:25.718868 12135 hybrid_clock.cc:648] HybridClock initialized: now 1781907985718848 us; error 30 us; skew 500 ppm
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:25.721702 12135 init.cc:377] Logged in from keytab as oryx/127.5.176.194@KRBTEST.COM (short username oryx)
I20260619 22:26:25.722688 12135 webserver.cc:460] Webserver started at http://127.5.176.194:33855/ using document root <none> and password file <none>
I20260619 22:26:25.723271 12135 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:25.723364 12135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:25.723600 12135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:25.725682 12135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "e8e2d6849ae3cdb59c616c281feb2541"
server_key_iv: "937253a6d28e92525641291604e7f56e"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.726356 12135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "e8e2d6849ae3cdb59c616c281feb2541"
server_key_iv: "937253a6d28e92525641291604e7f56e"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.730403 12135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:25.733207 12150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.734773 12135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:25.734936 12135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "e8e2d6849ae3cdb59c616c281feb2541"
server_key_iv: "937253a6d28e92525641291604e7f56e"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.735105 12135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:25.750600 12135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:25.753669 12135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:25.753891 12135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:25.754597 12135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:25.755435 12135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:25.755508 12135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.755579 12135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:25.755630 12135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.764936 12135 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:34737
I20260619 22:26:25.764953 12263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:34737 every 8 connection(s)
I20260619 22:26:25.765899 12135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:25.774788 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12135
I20260619 22:26:25.774938 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:25.775265 5827 external_mini_cluster.cc:1428] Setting key c2c8fcaeb0c9e79fb64b460235c10f6b
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:25.780443 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.767811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:60825 (local address 127.5.176.254:41209)
0619 22:26:25.767970 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:25.767975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:25.769097 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 22:26:25.769988 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:25.769991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:25.770039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:25.770120 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:25.771410 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.771937 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.773070 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.773284 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.773374 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:25.776215 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:25.776218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:25.776220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:25.776243 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:25.777942 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:25.778614 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:25.778620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:25.778623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:25.778725 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:25.779103 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:25.779108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:25.779111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:25.779376 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:25.779493 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:25.780043 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 22:26:25.780196 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:26:25.781198 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.768259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:60825)
0619 22:26:25.768946 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:25.768986 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:25.769782 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:25.770382 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:25.770389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:25.770758 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:25.771271 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:25.771284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.772066 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.772069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:25.772958 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:25.772967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.773461 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.773464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:25.773623 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:25.774289 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:25.774305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:25.776054 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:25.778054 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:25.778063 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:25.778082 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:25.778498 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:25.778854 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:25.778856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:25.778858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:25.778997 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:25.779516 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:25.779521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:25.779780 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 22:26:25.779941 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 22:26:25.780215 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":113,"threads_started":1}
I20260619 22:26:25.782414 12264 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41209
I20260619 22:26:25.782663 12264 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:25.783154 12264 heartbeater.cc:511] Master 127.5.176.254:41209 requested a full tablet report, sending...
I20260619 22:26:25.784178 11939 ts_manager.cc:194] Registered new tserver with Master: fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737)
I20260619 22:26:25.784813 11939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:60825
WARNING: no policy specified for oryx/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab.
Entry for principal oryx/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab.
I20260619 22:26:25.803508 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41209
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:25.914224 12269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:25.914489 12269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:25.914533 12269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:25.918418 12269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:25.918467 12269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:25.918524 12269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:25.923631 12269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:41209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:25.924613 12269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:25.925565 12269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:25.934948 12277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:25.935014 12274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.934978 12269 server_base.cc:1034] running on GCE node
W20260619 22:26:25.934953 12275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:25.935803 12269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:25.936455 12269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:25.937656 12269 hybrid_clock.cc:648] HybridClock initialized: now 1781907985937633 us; error 62 us; skew 500 ppm
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:25.943722 12269 init.cc:377] Logged in from keytab as oryx/127.5.176.195@KRBTEST.COM (short username oryx)
I20260619 22:26:25.944746 12269 webserver.cc:460] Webserver started at http://127.5.176.195:41759/ using document root <none> and password file <none>
I20260619 22:26:25.945348 12269 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:25.945444 12269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:25.945683 12269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:25.947907 12269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "7c7454088ed841fcb219b1e246cab7e1"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "119aa669f35d2844670ddf791c4743a9"
server_key_iv: "a663ac6c726a30d96e29b9533d35e264"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.948501 12269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "7c7454088ed841fcb219b1e246cab7e1"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "119aa669f35d2844670ddf791c4743a9"
server_key_iv: "a663ac6c726a30d96e29b9533d35e264"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.952414 12269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:25.954952 12284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.956236 12269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:25.956384 12269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "7c7454088ed841fcb219b1e246cab7e1"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "119aa669f35d2844670ddf791c4743a9"
server_key_iv: "a663ac6c726a30d96e29b9533d35e264"
server_key_version: "encryptionkey@0"
I20260619 22:26:25.956509 12269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:25.976301 12269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:25.979923 12269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:25.980147 12269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:25.980784 12269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:25.981637 12269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:25.981710 12269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.981782 12269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:25.981832 12269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:25.991074 12269 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:40835
I20260619 22:26:25.991096 12397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:40835 every 8 connection(s)
I20260619 22:26:25.992028 12269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:26.000134 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12269
I20260619 22:26:26.000288 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:26.000581 5827 external_mini_cluster.cc:1428] Setting key 3bb08c43d977026e4d27f553366d6983
Jun 19 22:26:25 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:26.005898 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.993857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:32843 (local address 127.5.176.254:41209)
0619 22:26:25.994018 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:25.994022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:25.995119 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 22:26:25.996028 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:25.996032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:25.996091 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:25.996197 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:25.997363 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.997898 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.999131 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.999352 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.999478 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.002306 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.002309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.002312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.002346 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.003762 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.004289 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.004293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.004295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.004375 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.004818 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.004821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.004823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.005001 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.005107 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.005601 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.005731 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:26:26.006740 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:25.994289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:32843)
0619 22:26:25.994962 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:25.995002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:25.995828 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:25.996386 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:25.996393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:25.996751 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:25.997218 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:25.997230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.998030 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.998034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:25.999002 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:25.999011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:25.999535 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:25.999539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:25.999701 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.000369 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.000403 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.002073 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.003895 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.003901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.003917 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.004174 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.004497 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.004500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.004502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.004711 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.005103 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.005111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.005365 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.005550 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.005873 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":115,"threads_started":1}
I20260619 22:26:26.007781 12398 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41209
I20260619 22:26:26.008006 12398 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:26.008445 12398 heartbeater.cc:511] Master 127.5.176.254:41209 requested a full tablet report, sending...
I20260619 22:26:26.009485 11939 ts_manager.cc:194] Registered new tserver with Master: 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835)
I20260619 22:26:26.010049 11939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:32843
I20260619 22:26:26.015159 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.253.keytab.
Entry for principal oryx/127.5.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.253.keytab.
I20260619 22:26:26.050354 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:44821
--webserver_interface=127.5.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.253.keytab
--principal=oryx/127.5.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:41209,127.5.176.253:44821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:26.164660 12403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:26.165002 12403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:26.165083 12403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:26.169289 12403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:26.169392 12403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:26.169430 12403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:26.169468 12403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:26.169505 12403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:26.174999 12403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:41209,127.5.176.253:44821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.253.keytab
--principal=oryx/127.5.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:44821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
W20260619 22:26:26.175305 12403 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.176.254:41209,127.5.176.253:44821'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 22:26:26.176393 12403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:26.177506 12403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:26.185978 12408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.185978 12411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.185985 12409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:26.186316 12403 server_base.cc:1034] running on GCE node
I20260619 22:26:26.186955 12403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:26.188050 12403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:26.189253 12403 hybrid_clock.cc:648] HybridClock initialized: now 1781907986189224 us; error 40 us; skew 500 ppm
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:26.192106 12403 init.cc:377] Logged in from keytab as oryx/127.5.176.253@KRBTEST.COM (short username oryx)
I20260619 22:26:26.193095 12403 webserver.cc:460] Webserver started at http://127.5.176.253:45361/ using document root <none> and password file <none>
I20260619 22:26:26.193655 12403 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.193751 12403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.193980 12403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:26.196077 12403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/instance:
uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "7b18de8d6e7e8c403980886c3192ac30"
server_key_iv: "f64341902901fdf6934b0a2e7c6efec7"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.196632 12403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal/instance:
uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "7b18de8d6e7e8c403980886c3192ac30"
server_key_iv: "f64341902901fdf6934b0a2e7c6efec7"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.200296 12403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:26:26.202665 12418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.203873 12403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:26.204017 12403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "7b18de8d6e7e8c403980886c3192ac30"
server_key_iv: "f64341902901fdf6934b0a2e7c6efec7"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.204125 12403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.228013 12403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:26.231416 12403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:26.231614 12403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:26.238819 12403 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.253:44821
I20260619 22:26:26.239152 12470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.253:44821 every 8 connection(s)
I20260619 22:26:26.239895 12403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
I20260619 22:26:26.243003 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:26.246899 12471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:26:26.247664 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12403
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.253@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
I20260619 22:26:26.262359 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.249055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.5.176.254:41209)
0619 22:26:26.249199 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.249203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.250291 (+ 1088us) server_negotiation.cc:407] Connection header received
0619 22:26:26.251212 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.251216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.251278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.251367 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.252588 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.253128 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.254394 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.254746 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.254855 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.258048 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.258054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.258058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.258096 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.260000 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.260633 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.260638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.260640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.260746 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.261091 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.261096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.261098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.261369 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.261473 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.262038 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.262201 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:26:26.263574 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.249242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.5.176.253:44821)
0619 22:26:26.249866 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.249883 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.249921 (+ 38us) server_negotiation.cc:407] Connection header received
0619 22:26:26.250925 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.250948 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.251197 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.251382 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.252405 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.253744 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.254816 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.255203 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.255328 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.258536 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.258541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.258552 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.258599 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.260688 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.261165 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.261171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.261173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.261249 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.261550 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.261553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.261555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.261808 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.262031 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.262275 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.262713 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":144,"threads_started":1}
I20260619 22:26:26.263574 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.249242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:45886)
0619 22:26:26.250049 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.250175 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.250952 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.251554 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.251560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.252040 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.252433 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.252454 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.253273 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.253277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.254272 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.254283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.255020 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.255023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.255209 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.255888 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.255907 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.257885 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.260115 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.260123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.260133 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.260482 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.260857 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.260859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.260861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.260990 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.261478 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.261484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.261792 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.262504 (+ 712us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.262769 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":49,"spinlock_wait_cycles":33536,"thread_start_us":154,"threads_started":1}
I20260619 22:26:26.265053 12471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.253@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
I20260619 22:26:26.274056 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.265614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.5.176.253:44821)
0619 22:26:26.265726 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.265730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.265743 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:26.265828 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.265830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.265869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.265949 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.266388 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.267228 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.268219 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.268531 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.268640 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.271007 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.271011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.271012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.271032 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.272447 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.272895 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.272897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.272898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.272972 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.273221 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.273223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.273225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.273397 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.273473 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.273807 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.273902 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 22:26:26.274058 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.265551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:58648)
0619 22:26:26.265680 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.265693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.265762 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.265992 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.265995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.266185 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.266298 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.266303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.267331 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.267334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.268115 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.268128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.268636 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.268639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.268726 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.269231 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.269245 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.270894 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.272562 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.272569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.272571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.272756 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.273063 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.273065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.273066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.273141 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.273505 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.273507 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.273588 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.273686 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.273800 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 22:26:26.277529 12471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Bootstrap starting.
I20260619 22:26:26.279881 12471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:26.280623 12471 log.cc:824] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:26.282644 12471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: No bootstrap required, opened a new log
I20260619 22:26:26.285763 12471 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } }
I20260619 22:26:26.285991 12471 raft_consensus.cc:374] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:26.286024 12471 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56a7b2ea9224e5dbfaf58dad68b0c96, State: Initialized, Role: FOLLOWER
I20260619 22:26:26.286506 12471 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } }
I20260619 22:26:26.287034 12477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } } }
I20260619 22:26:26.287173 12477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:26.287735 12471 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:26:26.292578 12488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:26:26.292672 12488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:26:26.294593 12403 master_runner.cc:428] Detected that this master e56a7b2ea9224e5dbfaf58dad68b0c96 is joining an existing cluster
I20260619 22:26:26.294669 12403 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.176.253:44821
I20260619 22:26:26.305476 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.296839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.5.176.254:41209)
0619 22:26:26.297090 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.297096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.297449 (+ 353us) server_negotiation.cc:407] Connection header received
0619 22:26:26.297594 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.297596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.297642 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.297734 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.298343 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.299128 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.300345 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.300629 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.300742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.301953 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.301957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.301960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.301986 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.303764 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304247 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.304250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.304252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.304348 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304629 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.304633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.304635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.304873 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.304969 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.305205 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.305318 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260619 22:26:26.305509 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.296935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:45896)
0619 22:26:26.297354 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.297369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.297488 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.297751 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.297754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.298034 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.298205 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.298214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.299245 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.299251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.300221 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.300232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.300741 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.300745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.300842 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.301419 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.301433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.301809 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.303881 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.303887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.303892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.304161 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.304454 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.304460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.304549 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.304952 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.304955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.305039 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.305138 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.305273 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260619 22:26:26.305853 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.296966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:58652)
0619 22:26:26.297683 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.297695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.297810 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.298055 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.298058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.298313 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.298467 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.298474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.299400 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.299403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.299974 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.299981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.300470 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.300474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.300565 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.301849 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.301863 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.302285 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.304197 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.304203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.304464 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.304692 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.304700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.304776 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.305174 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.305177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.305250 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.305528 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.305660 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":24,"thread_start_us":97,"threads_started":1}
I20260619 22:26:26.306046 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.296842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.5.176.253:44821)
0619 22:26:26.296971 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.296976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.297763 (+ 787us) server_negotiation.cc:407] Connection header received
0619 22:26:26.297889 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.297894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.297938 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.298019 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.298587 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.299280 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.300084 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.300348 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.300443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.302432 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.302437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.302441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.302467 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.304093 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304550 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.304554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.304555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.304622 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.304875 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.304878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.304880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.305052 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.305142 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.305411 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.305500 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.254@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
I20260619 22:26:26.319728 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.310504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.5.176.253:44821)
0619 22:26:26.310615 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.310619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.310988 (+ 369us) server_negotiation.cc:407] Connection header received
0619 22:26:26.311165 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.311168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.311217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.311298 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.311860 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.312593 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.313437 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.313771 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.313870 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.315933 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.315937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.315940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.315965 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.317727 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.318331 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.318337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.318342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.318427 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.318726 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.318732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.318736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.318967 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.319068 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.319466 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.319576 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:26:26.319764 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.310504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:58654)
0619 22:26:26.310863 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.310912 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.311040 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.311342 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.311346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.311663 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.311757 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.311762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.312698 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.312701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.313338 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.313347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.313874 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.313876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.313949 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.314483 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.314494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.315808 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.317850 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.317862 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.317871 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.318207 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.318524 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.318530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.318535 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.318628 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.319129 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.319139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.319258 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.319416 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.319604 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260619 22:26:26.321141 11939 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.176.253:44821: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
member_type: NON_VOTER
last_known_addr {
host: "127.5.176.253"
port: 44821
}
attrs {
promote: true
}
}
dest_uuid: "f5262501ae174235a345e305a6749db8"
cas_config_opid_index: -1
I20260619 22:26:26.321827 11939 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } }
I20260619 22:26:26.323879 12499 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } }
I20260619 22:26:26.324239 12445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:26.325713 12445 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Refusing update from remote peer f5262501ae174235a345e305a6749db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 22:26:26.326229 12477 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:26.326299 12499 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.176.253:44821
I20260619 22:26:26.326393 12477 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } }
I20260619 22:26:26.326728 12403 master.cc:537] Master@127.5.176.253:44821 shutting down...
I20260619 22:26:26.327564 12477 leader_election.cc:288] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209)
I20260619 22:26:26.327529 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } }
I20260619 22:26:26.327629 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.5.176.252@KRBTEST.COM; defaulting to no policy
I20260619 22:26:26.328248 11949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5262501ae174235a345e305a6749db8" is_pre_election: true
I20260619 22:26:26.328975 12422 leader_election.cc:302] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e56a7b2ea9224e5dbfaf58dad68b0c96; no voters: f5262501ae174235a345e305a6749db8
I20260619 22:26:26.329602 12477 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:26:26.329718 12500 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.252@KRBTEST.COM" created.
W20260619 22:26:26.332630 11924 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 22:26:26.333541 11924 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 -> Peer e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253:44821): Couldn't send request to peer e56a7b2ea9224e5dbfaf58dad68b0c96. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 22:26:26.334286 12403 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:26.334568 12403 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:26.334666 12403 tablet_replica.cc:330] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: stopping tablet replica
I20260619 22:26:26.339421 12403 master.cc:552] Master@127.5.176.253:44821 shutdown complete.
I20260619 22:26:26.339496 12403 master_runner.cc:305] Clearing existing system tablet
I20260619 22:26:26.339730 12403 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.339802 12403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.340965 12403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 22:26:26.341828 12505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.342036 12403 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.342191 12403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:26.342240 12403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "7b18de8d6e7e8c403980886c3192ac30"
server_key_iv: "f64341902901fdf6934b0a2e7c6efec7"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.344286 12403 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:26.347858 12403 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 22:26:26.347980 12403 log.cc:1196] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 22:26:26.348273 12403 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Deleting consensus metadata
I20260619 22:26:26.348527 12403 master_runner.cc:315] Copying system tablet from 127.5.176.254:41209
I20260619 22:26:26.349915 12403 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: tablet copy: Beginning tablet copy session from remote peer at address 127.5.176.254:41209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.252.keytab.
Entry for principal oryx/127.5.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.252.keytab.
I20260619 22:26:26.351256 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:37075
--webserver_interface=127.5.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.252.keytab
--principal=oryx/127.5.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:26:26.359776 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.350572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:45906)
0619 22:26:26.350845 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.350861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.350984 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.352927 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.352930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.353121 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.353223 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.353229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.354005 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.354010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.354691 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.354700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.355326 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.355329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.355401 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.356066 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.356079 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.356413 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.358174 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.358178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.358180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.358394 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.358684 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.358687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.358689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.358776 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.359147 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.359151 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.359294 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.359404 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.359591 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20260619 22:26:26.359764 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.352535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.5.176.254:41209)
0619 22:26:26.352673 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.352678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.352702 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:26:26.352744 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.352746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.352785 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.352891 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.353354 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.353888 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.354834 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.355068 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.355191 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.356562 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.356566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.356569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.356602 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.358020 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.358497 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.358500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.358502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.358585 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.358874 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.358877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.358879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.359053 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.359159 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.359473 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.359590 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:26:26.360668 11959 tablet_copy_service.cc:140] P f5262501ae174235a345e305a6749db8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e56a7b2ea9224e5dbfaf58dad68b0c96 ({username='oryx', principal='oryx/127.5.176.253@KRBTEST.COM'} at 127.0.0.1:45906)
I20260619 22:26:26.360828 11959 tablet_copy_service.cc:161] P f5262501ae174235a345e305a6749db8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e56a7b2ea9224e5dbfaf58dad68b0c96 at {username='oryx', principal='oryx/127.5.176.253@KRBTEST.COM'} at 127.0.0.1:45906: session id = e56a7b2ea9224e5dbfaf58dad68b0c96-00000000000000000000000000000000
I20260619 22:26:26.362284 11959 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 22:26:26.363732 12403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 22:26:26.365689 12403 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: tablet copy: Starting download of 0 data blocks...
I20260619 22:26:26.365892 12403 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: tablet copy: Starting download of 1 WAL segments...
I20260619 22:26:26.367534 12403 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 22:26:26.369097 11959 tablet_copy_service.cc:342] P f5262501ae174235a345e305a6749db8: Request end of tablet copy session e56a7b2ea9224e5dbfaf58dad68b0c96-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.176.253@KRBTEST.COM'} at 127.0.0.1:45906
I20260619 22:26:26.369204 11959 tablet_copy_service.cc:433] P f5262501ae174235a345e305a6749db8: ending tablet copy session e56a7b2ea9224e5dbfaf58dad68b0c96-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e56a7b2ea9224e5dbfaf58dad68b0c96
I20260619 22:26:26.375954 12403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:26.377934 12513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.377934 12514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.382407 12516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:26.383703 12403 server_base.cc:1034] running on GCE node
I20260619 22:26:26.383901 12403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:26.384266 12403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:26.385409 12403 hybrid_clock.cc:648] HybridClock initialized: now 1781907986385435 us; error 62 us; skew 500 ppm
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:26.386947 12403 init.cc:377] Logged in from keytab as oryx/127.5.176.253@KRBTEST.COM (short username oryx)
I20260619 22:26:26.387424 12403 webserver.cc:460] Webserver started at http://127.5.176.253:40325/ using document root <none> and password file <none>
I20260619 22:26:26.387611 12403 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.387665 12403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.388783 12403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:26.389729 12523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.389971 12403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:26.390069 12403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "7b18de8d6e7e8c403980886c3192ac30"
server_key_iv: "f64341902901fdf6934b0a2e7c6efec7"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.390203 12403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.406751 12403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:26.408715 12403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:26.413551 12403 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.253:44821
I20260619 22:26:26.414110 12403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
I20260619 22:26:26.414572 12575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.253:44821 every 8 connection(s)
I20260619 22:26:26.415688 12576 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:26:26.416610 12576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Bootstrap starting.
I20260619 22:26:26.424748 12576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:26:26.425081 12576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: Bootstrap complete.
I20260619 22:26:26.425581 12576 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } }
I20260619 22:26:26.425717 12576 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56a7b2ea9224e5dbfaf58dad68b0c96, State: Initialized, Role: LEARNER
I20260619 22:26:26.425830 12576 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } }
I20260619 22:26:26.426254 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } }
I20260619 22:26:26.426350 12576 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:26.426429 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:26:26.429315 12590 catalog_manager.cc:1259] Loaded cluster ID: f1068e962853462ba604b42122c242c4
I20260619 22:26:26.429425 12590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: loading cluster ID for follower catalog manager: success
I20260619 22:26:26.431370 12590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: acquiring CA information for follower catalog manager: success
I20260619 22:26:26.432606 12590 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 22:26:26.504287 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:26.504604 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:26.504705 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:26.509387 12512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:26.509461 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:26.509481 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:26.509498 12512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:26.509514 12512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:26.515671 12512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.252.keytab
--principal=oryx/127.5.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:37075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:26.516794 12512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:26.517882 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:26.526477 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:26.526609 12512 server_base.cc:1034] running on GCE node
W20260619 22:26:26.526497 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.526546 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:26.527411 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:26.528661 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:26.529865 12512 hybrid_clock.cc:648] HybridClock initialized: now 1781907986529849 us; error 28 us; skew 500 ppm
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:26.532786 12512 init.cc:377] Logged in from keytab as oryx/127.5.176.252@KRBTEST.COM (short username oryx)
I20260619 22:26:26.533625 12512 webserver.cc:460] Webserver started at http://127.5.176.252:35119/ using document root <none> and password file <none>
I20260619 22:26:26.534200 12512 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.534281 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.534452 12512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:26.536546 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/instance:
uuid: "0dd44d2bda484738a44e935e0f8a7841"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "c947804f271bc32b7d8650485a8c29e2"
server_key_iv: "f47e278ee4f6f95f1953335ff5d0abb9"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.537091 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal/instance:
uuid: "0dd44d2bda484738a44e935e0f8a7841"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "c947804f271bc32b7d8650485a8c29e2"
server_key_iv: "f47e278ee4f6f95f1953335ff5d0abb9"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.541191 12512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:26:26.543871 12605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.545298 12512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:26:26.545447 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
uuid: "0dd44d2bda484738a44e935e0f8a7841"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "c947804f271bc32b7d8650485a8c29e2"
server_key_iv: "f47e278ee4f6f95f1953335ff5d0abb9"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.545578 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.562331 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:26.565472 12512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:26.565685 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:26.567521 12130 heartbeater.cc:503] Master 127.5.176.254:41209 was elected leader, sending a full tablet report...
I20260619 22:26:26.573282 12512 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.252:37075
I20260619 22:26:26.573583 12657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.252:37075 every 8 connection(s)
I20260619 22:26:26.574676 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
I20260619 22:26:26.577491 12658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:26.578861 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12512
I20260619 22:26:26.581828 12658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.176.252@KRBTEST.COM
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.252@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:26.593626 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.583632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.5.176.254:41209)
0619 22:26:26.583746 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.583751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.583892 (+ 141us) server_negotiation.cc:407] Connection header received
0619 22:26:26.584010 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.584013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.584061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.584142 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.584772 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.585530 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.586794 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.587124 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.587242 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.589843 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.589847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.589850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.589884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.591483 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.592038 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.592041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.592043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.592151 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.592562 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.592565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.592566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.592782 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.592887 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.593295 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.593445 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:26.594678 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.580425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.5.176.252:37075)
0619 22:26:26.580866 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.580892 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.580927 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:26:26.581553 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.581569 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.581887 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.582243 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.583050 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.584399 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.585535 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.586278 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.586432 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.589178 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.589200 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.589219 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.589244 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.591526 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.592045 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.592050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.592052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.592192 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.592535 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.592540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.592542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.593024 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.593178 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.593437 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.594289 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":672,"server-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
I20260619 22:26:26.594678 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.583426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:45910)
0619 22:26:26.583760 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.583806 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.583919 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.584202 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.584206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.584533 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.584643 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.584648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.585657 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.585661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.586648 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.586655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.587305 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.587309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.587393 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.587969 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.587986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.589621 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.591609 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.591619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.591629 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.591943 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.592283 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.592287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.592289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.592465 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.592895 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.592901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.593040 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.593244 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.593605 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260619 22:26:26.596768 12658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.252@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
I20260619 22:26:26.605077 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.597252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:58660)
0619 22:26:26.597370 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.597385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.597453 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.597805 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.597807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.597939 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.598032 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.598036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.598998 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.599001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.599556 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.599564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.600030 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.600033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.600112 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.600627 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.600638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.601884 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.603703 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.603706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.603707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.603932 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.604227 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.604229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.604230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.604309 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.604664 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.604666 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.604745 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.604847 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.604953 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 22:26:26.605162 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.597367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.5.176.253:44821)
0619 22:26:26.597609 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.597612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.597623 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:26:26.597660 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.597662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.597708 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.597780 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.598126 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.598873 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.599678 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.599930 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.600020 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.602017 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.602020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.602022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.602045 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.603587 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.604045 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.604048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.604050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.604134 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.604388 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.604390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.604392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.604583 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.604661 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.604934 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.605039 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260619 22:26:26.605957 12658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.252@KRBTEST.COM for oryx/127.5.176.252@KRBTEST.COM
I20260619 22:26:26.613886 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.606472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.0.0.1:41060)
0619 22:26:26.606585 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.606601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.606671 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.606861 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.606863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.606988 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.607057 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.607061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.608154 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.608157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.608673 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.608680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.609225 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.609228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.609290 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.609831 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.609842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.611069 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.612572 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.612575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.612576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.612762 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.613054 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.613056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.613058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.613133 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.613511 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.613513 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.613586 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.613666 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.613766 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:26:26.613996 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.606551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.5.176.252:37075)
0619 22:26:26.606656 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.606659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.606670 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:26:26.606738 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.606740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.606772 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.606833 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.607169 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.607713 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.608815 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.609071 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.609150 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.611181 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.611185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.611188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.611207 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.612452 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.612866 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.612869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.612870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.612948 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.613245 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.613248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.613249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.613420 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.613503 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.613741 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.613828 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 22:26:26.617101 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Bootstrap starting.
I20260619 22:26:26.619439 12658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:26.620169 12658 log.cc:824] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:26.622263 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: No bootstrap required, opened a new log
I20260619 22:26:26.625252 12658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } }
I20260619 22:26:26.625449 12658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:26.625519 12658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd44d2bda484738a44e935e0f8a7841, State: Initialized, Role: FOLLOWER
I20260619 22:26:26.625948 12658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } }
I20260619 22:26:26.626536 12664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } } }
I20260619 22:26:26.626701 12664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:26.627331 12658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:26:26.631927 12675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:26:26.632020 12675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:26:26.636309 12512 master_runner.cc:428] Detected that this master 0dd44d2bda484738a44e935e0f8a7841 is joining an existing cluster
I20260619 22:26:26.636384 12512 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.176.252:37075
I20260619 22:26:26.651170 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.639184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.5.176.253:44821)
0619 22:26:26.639380 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.639385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.640359 (+ 974us) server_negotiation.cc:407] Connection header received
0619 22:26:26.640588 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.640592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.640649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.640722 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.641762 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.642484 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.643593 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.643948 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.644045 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.645457 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.645462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.645468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.645499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.647879 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.649444 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.649450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.649452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.649549 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.649861 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.649864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.649866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.650372 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.650445 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.650859 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.651007 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:26:26.652261 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.639268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:58666)
0619 22:26:26.640136 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.640152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.640468 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.640905 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.640908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.641155 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.641479 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.641487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.642590 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.642596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.643472 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.643481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.644094 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.644099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.644269 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.644928 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.644943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.645323 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.648378 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.648381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.648384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.649320 (+ 936us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.649667 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.649670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.649672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.649760 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.650515 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.650518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.650629 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.651921 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.652070 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":48,"spinlock_wait_cycles":768,"thread_start_us":117,"threads_started":1}
I20260619 22:26:26.652971 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12001
I20260619 22:26:26.659461 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.639731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.0.0.1:41068)
0619 22:26:26.641021 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.641035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.641144 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.641460 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.641465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.641900 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.642013 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.642020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.643618 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.643621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.644942 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.644953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.645513 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.645517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.645621 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.646380 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.646427 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.647560 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.649856 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.649859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.649860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.650269 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.650535 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.650537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.650539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.650612 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.652912 (+ 2300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.652915 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.658826 (+ 5911us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.658970 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.659193 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":53,"thread_start_us":99,"threads_started":1}
I20260619 22:26:26.659547 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.641132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.5.176.254:41209)
0619 22:26:26.641271 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.641276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.646612 (+ 5336us) server_negotiation.cc:407] Connection header received
0619 22:26:26.646728 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.646733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.646853 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.646944 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.651677 (+ 4733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.652475 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.654327 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.654649 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.654727 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.656084 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.656089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.656091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.656120 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.657784 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.658273 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.658277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.658279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.658372 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.658644 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.658647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.658648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.658843 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.658921 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.659246 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.659380 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:26.660051 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.640006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.5.176.252:37075)
0619 22:26:26.640101 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.640106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.641095 (+ 989us) server_negotiation.cc:407] Connection header received
0619 22:26:26.641230 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.641233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.641268 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.641351 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.642188 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.642994 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.645076 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.645399 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.645519 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.647704 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.647708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.647714 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.647742 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.649676 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.650369 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.650373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.650374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.650444 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.650697 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.650700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.650702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.650907 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.650974 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.659742 (+ 8768us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.659888 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36,"spinlock_wait_cycles":1408}
I20260619 22:26:26.660099 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.640613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:45918)
0619 22:26:26.646488 (+ 5875us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.646505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.646614 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.651091 (+ 4477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.651095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.651328 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.651479 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.651488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.652599 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.652604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.654045 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.654186 (+ 141us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.654884 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.654888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.654981 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.655514 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.655528 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.655927 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.657914 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.657916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.657919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.658169 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.658472 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.658474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.658476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.658554 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.658975 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.658977 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.659063 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.659709 (+ 646us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.659943 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5804,"mutex_wait_us":121,"thread_start_us":211,"threads_started":1}
I20260619 22:26:26.662488 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:39413
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:41209
--webserver_port=38233
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907985, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.254@KRBTEST.COM for oryx/127.5.176.252@KRBTEST.COM
I20260619 22:26:26.698374 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.670460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.0.0.1:41074)
0619 22:26:26.670641 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.670675 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.670799 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.671530 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.671534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.671816 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.671960 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.671972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.673143 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.673148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.674190 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.674205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.675017 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.675023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.675138 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.675713 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.675733 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.677480 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.696455 (+ 18975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.696460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.696463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.696787 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.697148 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.697151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.697153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.697254 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.697755 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.697758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.697869 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.697982 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.698179 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 22:26:26.698428 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.670942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.5.176.252:37075)
0619 22:26:26.671057 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.671062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.671077 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:26:26.671170 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.671172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.671242 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.671333 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.672126 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.672988 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.674351 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.674732 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.674856 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.677670 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.677675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.677680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.677717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.696248 (+ 18531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.696920 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.696925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.696927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.697035 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.697361 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.697366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.697369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.697634 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.697761 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.698063 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.698246 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:26:26.699968 11939 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.176.252:37075: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841"
member_type: NON_VOTER
last_known_addr {
host: "127.5.176.252"
port: 37075
}
attrs {
promote: true
}
}
dest_uuid: "f5262501ae174235a345e305a6749db8"
cas_config_opid_index: 5
I20260619 22:26:26.700598 11939 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } }
I20260619 22:26:26.701920 12499 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.703339 12499 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.176.252:37075
I20260619 22:26:26.703503 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.703596 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:26.704120 12512 master.cc:537] Master@127.5.176.252:37075 shutting down...
I20260619 22:26:26.704449 12632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:26.706307 12632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Refusing update from remote peer f5262501ae174235a345e305a6749db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 22:26:26.707302 12500 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 22:26:26.709339 11925 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 -> Peer 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252:37075): Couldn't send request to peer 0dd44d2bda484738a44e935e0f8a7841. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 22:26:26.712841 12512 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:26.712945 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.701890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.5.176.253:44821)
0619 22:26:26.702018 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.702023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.705610 (+ 3587us) server_negotiation.cc:407] Connection header received
0619 22:26:26.705894 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.705898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.705963 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.706031 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:26.707224 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.708013 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.711372 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.712072 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.712152 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.712310 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.712724 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.712808 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:26:26.713349 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.701789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:58670)
0619 22:26:26.705525 (+ 3736us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.705542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.705687 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.706813 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.706816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.706836 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:26.706929 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.706935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.708158 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.708161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.710989 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.711179 (+ 190us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.712465 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.712468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.712562 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.712608 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.713136 (+ 528us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.713233 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2901}
I20260619 22:26:26.713901 12550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 LEARNER]: Refusing update from remote peer f5262501ae174235a345e305a6749db8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 22:26:26.713593 12512 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:26.714278 12512 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: stopping tablet replica
I20260619 22:26:26.714408 12500 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 22:26:26.716414 12550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.718914 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5262501ae174235a345e305a6749db8. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.718935 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: NON_VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.719071 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:26:26.719029 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:26:26.719655 12500 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: attempting to promote NON_VOTER e56a7b2ea9224e5dbfaf58dad68b0c96 to VOTER
I20260619 22:26:26.720207 12500 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } }
I20260619 22:26:26.721861 12550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 LEARNER]: Refusing update from remote peer f5262501ae174235a345e305a6749db8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260619 22:26:26.721972 11925 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 -> Peer 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252:37075): Couldn't send request to peer 0dd44d2bda484738a44e935e0f8a7841. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 22:26:26.722910 12499 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 22:26:26.725098 12499 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.725621 12512 master.cc:552] Master@127.5.176.252:37075 shutdown complete.
I20260619 22:26:26.725700 12512 master_runner.cc:305] Clearing existing system tablet
I20260619 22:26:26.725999 12512 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.726081 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.726822 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.726943 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:26.727082 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.727165 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:26.728020 12512 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:26.727990 12550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.729713 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.729843 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:26.730402 12689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.730621 12512 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.730746 12512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260619 22:26:26.730780 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
uuid: "0dd44d2bda484738a44e935e0f8a7841"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "c947804f271bc32b7d8650485a8c29e2"
server_key_iv: "f47e278ee4f6f95f1953335ff5d0abb9"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.732787 12512 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:26.735131 12512 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 22:26:26.735216 12512 log.cc:1196] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 22:26:26.735451 12512 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Deleting consensus metadata
I20260619 22:26:26.735610 12512 master_runner.cc:315] Copying system tablet from 127.5.176.254:41209
I20260619 22:26:26.736770 12512 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: tablet copy: Beginning tablet copy session from remote peer at address 127.5.176.254:41209
I20260619 22:26:26.746322 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.737458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.5.176.254:41209)
0619 22:26:26.737585 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.737593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.737828 (+ 235us) server_negotiation.cc:407] Connection header received
0619 22:26:26.738027 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.738031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.738084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.738215 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.738814 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.739654 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.740826 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.741160 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.741275 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.742677 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.742681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.742684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.742715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.744450 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.744970 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.744974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.744976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.745063 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.745375 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.745378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.745380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.745592 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.745687 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.746020 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.746133 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 22:26:26.746361 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.737389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:45926)
0619 22:26:26.737717 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.737735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.737867 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.738273 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.738276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.738513 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.738678 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.738687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.739776 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.739780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.740692 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.740703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.741296 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.741300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.741402 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.741997 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.742014 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.742510 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.744572 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.744575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.744578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.744860 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.745169 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.745172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.745175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.745275 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.745700 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.745703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.745838 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.745957 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.746109 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
I20260619 22:26:26.747342 11959 tablet_copy_service.cc:140] P f5262501ae174235a345e305a6749db8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0dd44d2bda484738a44e935e0f8a7841 ({username='oryx', principal='oryx/127.5.176.252@KRBTEST.COM'} at 127.0.0.1:45926)
I20260619 22:26:26.747442 11959 tablet_copy_service.cc:161] P f5262501ae174235a345e305a6749db8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0dd44d2bda484738a44e935e0f8a7841 at {username='oryx', principal='oryx/127.5.176.252@KRBTEST.COM'} at 127.0.0.1:45926: session id = 0dd44d2bda484738a44e935e0f8a7841-00000000000000000000000000000000
I20260619 22:26:26.748531 11959 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 22:26:26.749887 12512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 22:26:26.752305 12512 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: tablet copy: Starting download of 0 data blocks...
I20260619 22:26:26.752485 12512 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: tablet copy: Starting download of 1 WAL segments...
I20260619 22:26:26.753921 12512 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 22:26:26.755652 11959 tablet_copy_service.cc:342] P f5262501ae174235a345e305a6749db8: Request end of tablet copy session 0dd44d2bda484738a44e935e0f8a7841-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.176.252@KRBTEST.COM'} at 127.0.0.1:45926
I20260619 22:26:26.755735 11959 tablet_copy_service.cc:433] P f5262501ae174235a345e305a6749db8: ending tablet copy session 0dd44d2bda484738a44e935e0f8a7841-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0dd44d2bda484738a44e935e0f8a7841
I20260619 22:26:26.763685 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:26.765446 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.770318 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.770499 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:26.774371 12512 server_base.cc:1034] running on GCE node
I20260619 22:26:26.774588 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:26.775024 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:26.776206 12512 hybrid_clock.cc:648] HybridClock initialized: now 1781907986776179 us; error 48 us; skew 500 ppm
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:26.777951 12512 init.cc:377] Logged in from keytab as oryx/127.5.176.252@KRBTEST.COM (short username oryx)
I20260619 22:26:26.778445 12512 webserver.cc:460] Webserver started at http://127.5.176.252:42363/ using document root <none> and password file <none>
I20260619 22:26:26.778636 12512 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.778680 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.780046 12512 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:26.781154 12706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.781407 12512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:26.781486 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
uuid: "0dd44d2bda484738a44e935e0f8a7841"
format_stamp: "Formatted at 2026-06-19 22:26:26 on dist-test-slave-bt27"
server_key: "c947804f271bc32b7d8650485a8c29e2"
server_key_iv: "f47e278ee4f6f95f1953335ff5d0abb9"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.781574 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.786940 12264 heartbeater.cc:503] Master 127.5.176.254:41209 was elected leader, sending a full tablet report...
I20260619 22:26:26.795238 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:26.797544 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:26.803006 12512 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.252:37075
I20260619 22:26:26.803049 12758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.252:37075 every 8 connection(s)
I20260619 22:26:26.803603 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
I20260619 22:26:26.805470 12759 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:26:26.806658 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Bootstrap starting.
W20260619 22:26:26.817021 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:26.817443 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:26.817539 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 22:26:26.819135 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:26:26.819473 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Bootstrap complete.
I20260619 22:26:26.820099 12759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } }
I20260619 22:26:26.820253 12759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0dd44d2bda484738a44e935e0f8a7841, State: Initialized, Role: LEARNER
I20260619 22:26:26.820367 12759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } }
I20260619 22:26:26.820734 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:26.820909 12759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:26.820917 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:26:26.823995 12776 catalog_manager.cc:1259] Loaded cluster ID: f1068e962853462ba604b42122c242c4
I20260619 22:26:26.824118 12776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: loading cluster ID for follower catalog manager: success
W20260619 22:26:26.824534 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:26.824631 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:26.824762 12684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:26.826442 12776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: acquiring CA information for follower catalog manager: success
I20260619 22:26:26.828294 12776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:26:26.831188 12684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:39413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=38233
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:26.832309 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:26.833334 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:26.843546 12779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.843479 12778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:26.843572 12781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:26.843962 12684 server_base.cc:1034] running on GCE node
I20260619 22:26:26.844328 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:26.844981 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:26.846210 12684 hybrid_clock.cc:648] HybridClock initialized: now 1781907986846187 us; error 31 us; skew 500 ppm
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:26.848981 12684 init.cc:377] Logged in from keytab as oryx/127.5.176.193@KRBTEST.COM (short username oryx)
I20260619 22:26:26.849833 12684 webserver.cc:460] Webserver started at http://127.5.176.193:38233/ using document root <none> and password file <none>
I20260619 22:26:26.850485 12684 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:26.850596 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:26.854364 12684 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 22:26:26.856429 12788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.857741 12684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:26.857883 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "60620add323f9707aa70b9fff3ff45d4"
server_key_iv: "ae763f4b455f3714660627e7b6154d36"
server_key_version: "encryptionkey@0"
I20260619 22:26:26.858496 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:26.880263 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:26.883695 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:26.883914 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:26.884965 12684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:26.885838 12684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:26.885908 12684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.885982 12684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:26.886025 12684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:26.895784 12684 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:39413
I20260619 22:26:26.895802 12901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:39413 every 8 connection(s)
I20260619 22:26:26.896797 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:26.901386 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12684
I20260619 22:26:26.901592 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12135
I20260619 22:26:26.908705 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:34737
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:41209
--webserver_port=33855
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
Jun 19 22:26:26 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907986, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.252@KRBTEST.COM
I20260619 22:26:26.915272 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.898592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:34677 (local address 127.5.176.254:41209)
0619 22:26:26.898734 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.898738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.899809 (+ 1071us) server_negotiation.cc:407] Connection header received
0619 22:26:26.900925 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.900932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.900994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.901081 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.903786 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.904639 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.906615 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.907009 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.907105 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.910410 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.910413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.910415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.910439 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.911905 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.912973 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.912977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.912979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.913068 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.913410 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.913414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.913415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.913671 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.913774 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.914970 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.915098 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:26:26.916330 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.899030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:34677)
0619 22:26:26.899647 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.899709 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.900692 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.901287 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.901294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.902820 (+ 1526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.903597 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.903620 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.904823 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.904826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.906470 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.906484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.907311 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.907318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.907481 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.908344 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.908359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.910205 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.912063 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.912072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.912089 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.912855 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.913162 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.913166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.913168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.913307 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.913828 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.913834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.914454 (+ 620us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.915688 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.915826 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":169,"spinlock_wait_cycles":12032,"thread_start_us":131,"threads_started":1}
I20260619 22:26:26.917045 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.899221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:52173 (local address 127.5.176.253:44821)
0619 22:26:26.899351 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.899355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.900296 (+ 941us) server_negotiation.cc:407] Connection header received
0619 22:26:26.900963 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.900966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.901025 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.901081 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.904063 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.904842 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.906565 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.906906 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.906986 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.911632 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.911638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.911644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.911679 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.913452 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.913926 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.913930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.913931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.914015 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.914360 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.914363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.914365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.914577 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.914669 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.916766 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.916885 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:26:26.916612 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.899030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.5.176.193:52173)
0619 22:26:26.900197 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.900208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.900718 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.901284 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.901291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.903342 (+ 2051us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.903933 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.903940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.904967 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.904971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.906428 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.906438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.907158 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.907161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.907471 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.908269 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.908305 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.911456 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.913568 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.913571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.913573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.913823 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.914121 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.914125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.914128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.914246 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.914715 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.914719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.914835 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.914947 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.915276 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":1842,"spinlock_wait_cycles":4608,"thread_start_us":112,"threads_started":1}
I20260619 22:26:26.917784 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.899030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.5.176.193:58949)
0619 22:26:26.899647 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:26.899709 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:26.900692 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:26.903079 (+ 2387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:26.903082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:26.903251 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:26.903664 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.903675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.905185 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.905188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.906428 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:26.906438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.907158 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.907161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:26.907600 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.908285 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:26.908306 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:26.912666 (+ 4360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:26.915833 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.915839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.915844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.916113 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.916477 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:26.916482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:26.916485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:26.916579 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:26.917112 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:26.917117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:26.917238 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 22:26:26.917358 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 22:26:26.917523 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":443,"spinlock_wait_cycles":31104,"thread_start_us":136,"threads_started":1}
I20260619 22:26:26.918193 12904 heartbeater.cc:347] Connected to a master server at 127.5.176.253:44821
I20260619 22:26:26.918434 12904 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:26.918505 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:26.899212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:58949 (local address 127.5.176.252:37075)
0619 22:26:26.899512 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:26.899516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:26.899807 (+ 291us) server_negotiation.cc:407] Connection header received
0619 22:26:26.901038 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:26.901041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:26.902860 (+ 1819us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:26.902954 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:26.903813 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.905061 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.906565 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:26.906907 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:26.906998 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:26.913737 (+ 6739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:26.913743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:26.913750 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:26.913787 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:26.915691 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.916255 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.916261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.916264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.916357 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:26.916694 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:26.916699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:26.916702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:26.916958 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:26.917060 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:26.918077 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0619 22:26:26.918268 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260619 22:26:26.918989 12904 heartbeater.cc:511] Master 127.5.176.253:44821 requested a full tablet report, sending...
I20260619 22:26:26.919144 12902 heartbeater.cc:347] Connected to a master server at 127.5.176.252:37075
I20260619 22:26:26.919256 12902 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:26.919464 12902 heartbeater.cc:511] Master 127.5.176.252:37075 requested a full tablet report, sending...
I20260619 22:26:26.919838 12903 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41209
I20260619 22:26:26.919919 12903 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:26.920073 12903 heartbeater.cc:511] Master 127.5.176.254:41209 requested a full tablet report, sending...
I20260619 22:26:26.920598 12536 ts_manager.cc:194] Registered new tserver with Master: f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413)
I20260619 22:26:26.920635 12723 ts_manager.cc:194] Registered new tserver with Master: f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413)
I20260619 22:26:26.920768 11939 ts_manager.cc:194] Re-registered known tserver with Master: f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413)
I20260619 22:26:26.921432 11939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:34677
I20260619 22:26:27.012099 12398 heartbeater.cc:503] Master 127.5.176.254:41209 was elected leader, sending a full tablet report...
W20260619 22:26:27.026129 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:27.026544 12910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:27.026629 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:27.031076 12910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:27.031185 12910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:27.031276 12910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:27.036969 12910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:34737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=33855
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:27.038259 12910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:27.039407 12910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:27.049486 12916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:27.049467 12915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:27.049463 12918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:27.050175 12910 server_base.cc:1034] running on GCE node
I20260619 22:26:27.050567 12910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:27.051185 12910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:27.052342 12910 hybrid_clock.cc:648] HybridClock initialized: now 1781907987052312 us; error 45 us; skew 500 ppm
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:27.055179 12910 init.cc:377] Logged in from keytab as oryx/127.5.176.194@KRBTEST.COM (short username oryx)
I20260619 22:26:27.056277 12910 webserver.cc:460] Webserver started at http://127.5.176.194:33855/ using document root <none> and password file <none>
I20260619 22:26:27.057054 12910 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:27.057129 12910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:27.061275 12910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 22:26:27.063517 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:27.064800 12910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:27.064919 12910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "e8e2d6849ae3cdb59c616c281feb2541"
server_key_iv: "937253a6d28e92525641291604e7f56e"
server_key_version: "encryptionkey@0"
I20260619 22:26:27.065506 12910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:27.098017 12910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:27.101107 12910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:27.101315 12910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:27.101985 12910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:27.102917 12910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:27.102969 12910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:27.103008 12910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:27.103024 12910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:27.112373 12910 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:34737
I20260619 22:26:27.112399 13038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:34737 every 8 connection(s)
I20260619 22:26:27.113448 12910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:27.116511 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 12910
I20260619 22:26:27.116695 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12269
I20260619 22:26:27.124043 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:40835
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:41209
--webserver_port=41759
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.252@KRBTEST.COM
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:27.133869 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.115858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:57593 (local address 127.5.176.253:44821)
0619 22:26:27.116113 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.116119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.118855 (+ 2736us) server_negotiation.cc:407] Connection header received
0619 22:26:27.119426 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.119430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.119477 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.119530 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.120798 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.121616 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.123879 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.124231 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.124363 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.128655 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.128660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.128667 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.128707 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.131034 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.131781 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.131785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.131786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.131894 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132381 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.132387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.132392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.132641 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.132720 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.133561 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.133696 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260619 22:26:27.133869 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.115981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:36855 (local address 127.5.176.252:37075)
0619 22:26:27.116230 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.116237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.119100 (+ 2863us) server_negotiation.cc:407] Connection header received
0619 22:26:27.119299 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.119303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.119362 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.119434 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.120795 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.121634 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.123892 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.124149 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.124276 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.128971 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.128980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.128984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.129021 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.131432 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132061 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.132066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.132068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.132145 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132453 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.132456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.132457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.132640 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.132720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.133561 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.133696 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260619 22:26:27.133958 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.115699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:33643 (local address 127.5.176.254:41209)
0619 22:26:27.116204 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.116208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.116745 (+ 537us) server_negotiation.cc:407] Connection header received
0619 22:26:27.118626 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.118633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.118700 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.118837 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.120789 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.121693 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.125287 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.125707 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.125841 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.129218 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.129226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.129231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.129270 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.131600 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132253 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.132260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.132264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.132374 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132722 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.132727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.132730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.133031 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.133132 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.133585 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.133733 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260619 22:26:27.135879 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.116069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.5.176.194:57593)
0619 22:26:27.118767 (+ 2698us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.118783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.119312 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.119671 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.119675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.119891 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:27.120653 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.120660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.121847 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.121851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.123734 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.123745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.124532 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.124538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.124666 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.125995 (+ 1329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:27.126019 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:27.128152 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:27.131216 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.131225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.131248 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.131658 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.132011 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.132021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.132275 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.132789 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:27.132796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:27.133242 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.134097 (+ 855us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.134607 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2575,"mutex_wait_us":1379,"spinlock_wait_cycles":18944,"thread_start_us":86,"threads_started":1}
I20260619 22:26:27.135905 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.116069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.5.176.194:36855)
0619 22:26:27.119025 (+ 2956us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.119039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.119173 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.119545 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.119548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.119858 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:27.120605 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.120653 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.121847 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.121851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.123734 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.123745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.124325 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.124329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.124532 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.125395 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:27.125423 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:27.128653 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:27.131612 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.131615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.131617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.131945 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.132253 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.132260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.132356 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.132789 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:27.132796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:27.133242 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.134097 (+ 855us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.134655 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2856,"mutex_wait_us":1803,"spinlock_wait_cycles":31872,"thread_start_us":101,"threads_started":1}
I20260619 22:26:27.136688 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.116069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:33643)
0619 22:26:27.116583 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.116645 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.118354 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.119163 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.119174 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.119787 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:27.120601 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.120623 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.121859 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.121865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.125124 (+ 3259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.125137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.125861 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.125865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.125955 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.126516 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:27.126536 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:27.129005 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:27.131750 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.131755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.131757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.132119 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.132503 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.132508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.132512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.132606 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.133200 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:27.133204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:27.133339 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.134834 (+ 1495us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.135020 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":1735,"spinlock_wait_cycles":1920,"thread_start_us":130,"threads_started":1}
I20260619 22:26:27.137398 13039 heartbeater.cc:347] Connected to a master server at 127.5.176.252:37075
I20260619 22:26:27.137650 13039 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:27.137398 13041 heartbeater.cc:347] Connected to a master server at 127.5.176.253:44821
I20260619 22:26:27.137840 13041 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:27.137869 13040 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41209
I20260619 22:26:27.137960 13040 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:27.138501 13039 heartbeater.cc:511] Master 127.5.176.252:37075 requested a full tablet report, sending...
I20260619 22:26:27.138501 13040 heartbeater.cc:511] Master 127.5.176.254:41209 requested a full tablet report, sending...
I20260619 22:26:27.139202 13041 heartbeater.cc:511] Master 127.5.176.253:44821 requested a full tablet report, sending...
I20260619 22:26:27.139793 12723 ts_manager.cc:194] Registered new tserver with Master: fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737)
I20260619 22:26:27.139806 11939 ts_manager.cc:194] Re-registered known tserver with Master: fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737)
I20260619 22:26:27.140529 12536 ts_manager.cc:194] Registered new tserver with Master: fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737)
I20260619 22:26:27.140616 11939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:33643
W20260619 22:26:27.247400 13046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:27.247758 13046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:27.247810 13046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:27.251933 13046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:27.251984 13046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:27.252074 13046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:27.257405 13046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:40835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=41759
--tserver_master_addrs=127.5.176.254:41209,127.5.176.253:44821,127.5.176.252:37075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:27.258594 13046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:27.259644 13046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:27.269871 13052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:27.269944 13046 server_base.cc:1034] running on GCE node
W20260619 22:26:27.269860 13054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:27.269860 13051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:27.270615 13046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:27.271534 13046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:27.272748 13046 hybrid_clock.cc:648] HybridClock initialized: now 1781907987272696 us; error 39 us; skew 500 ppm
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:27.276158 13046 init.cc:377] Logged in from keytab as oryx/127.5.176.195@KRBTEST.COM (short username oryx)
I20260619 22:26:27.277099 13046 webserver.cc:460] Webserver started at http://127.5.176.195:41759/ using document root <none> and password file <none>
I20260619 22:26:27.277670 13046 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:27.277773 13046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:27.281852 13046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:27.284003 13061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:27.285252 13046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 22:26:27.285387 13046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "7c7454088ed841fcb219b1e246cab7e1"
format_stamp: "Formatted at 2026-06-19 22:26:25 on dist-test-slave-bt27"
server_key: "119aa669f35d2844670ddf791c4743a9"
server_key_iv: "a663ac6c726a30d96e29b9533d35e264"
server_key_version: "encryptionkey@0"
I20260619 22:26:27.285997 13046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:27.311069 13046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:27.312918 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.309159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.5.176.252:37075)
0619 22:26:27.309301 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.309306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.309441 (+ 135us) server_negotiation.cc:407] Connection header received
0619 22:26:27.309570 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.309573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.309637 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.309750 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:27.309993 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.310763 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.311923 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.312363 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.312485 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.312631 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.312719 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.312786 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:26:27.312918 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.309009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.0.0.1:41086)
0619 22:26:27.309346 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.309358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.309469 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.309785 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.309787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.309801 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:27.309889 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.309894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.310867 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.310869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.311799 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.311810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.312493 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.312497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.312582 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.312604 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.312697 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.312785 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260619 22:26:27.313488 12733 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 22:26:27.313963 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5262501ae174235a345e305a6749db8. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:27.314117 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: This master's current role is: LEARNER
I20260619 22:26:27.314894 13046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:27.315047 13046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:27.315568 13046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:27.316375 13046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:27.316419 13046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:27.316458 13046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:27.316474 13046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:27.325500 13046 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:40835
I20260619 22:26:27.325517 13176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:40835 every 8 connection(s)
I20260619 22:26:27.326687 13046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:27.331952 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 13046
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.253@KRBTEST.COM
Jun 19 22:26:27 dist-test-slave-bt27 krb5kdc[11894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907987, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.252@KRBTEST.COM
I20260619 22:26:27.346036 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.330502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:34907 (local address 127.5.176.254:41209)
0619 22:26:27.330651 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.330655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.331157 (+ 502us) server_negotiation.cc:407] Connection header received
0619 22:26:27.331250 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.331257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.331320 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.331409 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.334328 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.335113 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336222 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.336527 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336605 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.341682 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.341688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.341694 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.341747 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.343539 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344018 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.344021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.344023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.344098 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344394 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.344397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.344398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.345218 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.345321 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.345688 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.345843 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":585,"server-negotiator.queue_time_us":64}
I20260619 22:26:27.346294 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.328611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:43805 (local address 127.5.176.253:44821)
0619 22:26:27.328841 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.328847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.329658 (+ 811us) server_negotiation.cc:407] Connection header received
0619 22:26:27.331294 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.331297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.331353 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.331413 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.333254 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.333973 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.335815 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.336161 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336249 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.340688 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.340694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.340697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.340730 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.343233 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.343823 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.343826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.343828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.343914 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344335 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.344340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.344342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.344584 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.344660 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.345989 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.346106 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 22:26:27.346935 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.329068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.5.176.195:43805)
0619 22:26:27.329493 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.329537 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.331028 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.331594 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.331601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.332238 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:27.333066 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.333087 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.334121 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.334124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.335680 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.335690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.336434 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.336743 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.337612 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:27.337638 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:27.340473 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:27.343375 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.343379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.343381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.343704 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.344021 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.344025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.344227 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.344714 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:27.344719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:27.345112 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.346460 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.346570 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":307,"spinlock_wait_cycles":22016,"thread_start_us":103,"threads_started":1}
I20260619 22:26:27.347339 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.328683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:52461 (local address 127.5.176.252:37075)
0619 22:26:27.328830 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.328835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.330217 (+ 1382us) server_negotiation.cc:407] Connection header received
0619 22:26:27.331397 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.331400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.331473 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.331544 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.333266 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.333911 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.335847 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.336193 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336301 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.340977 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.340981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.340985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.341014 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.342999 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.343839 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.343844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.343849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.343939 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344346 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.344353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.344358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.344636 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.344748 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.347072 (+ 2324us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.347191 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:26:27.346939 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.329842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.5.176.195:52461)
0619 22:26:27.330112 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.330126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.331036 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.331951 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.331957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.332273 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:27.333067 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.333090 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.334094 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.334099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.335689 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.335701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.336443 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.336746 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.337707 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:27.337730 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:27.340813 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:27.343159 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.343168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.343190 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.343676 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.344040 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.344049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.344231 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.344797 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:27.344802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:27.345113 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.345335 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.345706 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":382,"spinlock_wait_cycles":47744,"thread_start_us":132,"threads_started":1}
I20260619 22:26:27.346933 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.330387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:34907)
0619 22:26:27.331033 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.331047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.331156 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.331594 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.331603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.332238 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:27.334178 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.334189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.335257 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.335260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.336096 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.336105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.336777 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.336780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.336870 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.337612 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:27.337633 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:27.341518 (+ 3885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:27.343661 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.343667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.343670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.343905 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.344203 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:27.344207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:27.344209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:27.344295 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:27.345371 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:27.345374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:27.345473 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.346315 (+ 842us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.346469 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":93,"spinlock_wait_cycles":751232,"thread_start_us":116,"threads_started":1}
I20260619 22:26:27.348322 13179 heartbeater.cc:347] Connected to a master server at 127.5.176.253:44821
I20260619 22:26:27.348359 13177 heartbeater.cc:347] Connected to a master server at 127.5.176.252:37075
I20260619 22:26:27.348521 13178 heartbeater.cc:347] Connected to a master server at 127.5.176.254:41209
I20260619 22:26:27.348557 13177 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:27.348603 13178 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:27.348557 13179 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:27.349134 13177 heartbeater.cc:511] Master 127.5.176.252:37075 requested a full tablet report, sending...
I20260619 22:26:27.349148 13179 heartbeater.cc:511] Master 127.5.176.253:44821 requested a full tablet report, sending...
I20260619 22:26:27.350315 12723 ts_manager.cc:194] Registered new tserver with Master: 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835)
I20260619 22:26:27.350359 12536 ts_manager.cc:194] Registered new tserver with Master: 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835)
I20260619 22:26:27.351053 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.334497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.5.176.252:37075)
0619 22:26:27.336635 (+ 2138us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.336640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.336656 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:27.336696 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.336698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.336751 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.336848 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.338857 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.339801 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.341182 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.341581 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.342100 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.343927 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.343931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.343934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.343953 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.345721 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.346649 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.346652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.346653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.346733 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.347079 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.347082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.347083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.347286 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.349301 (+ 2015us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.349373 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.349488 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2056,"thread_start_us":61,"threads_started":1}
I20260619 22:26:27.351385 13178 heartbeater.cc:511] Master 127.5.176.254:41209 requested a full tablet report, sending...
I20260619 22:26:27.352350 11939 ts_manager.cc:194] Re-registered known tserver with Master: 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835)
I20260619 22:26:27.353186 11939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:34907
I20260619 22:26:27.356225 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.333965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.5.176.253:44821)
0619 22:26:27.339215 (+ 5250us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.339220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.339236 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:27.339278 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.339281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.339329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.348920 (+ 9591us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.349035 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.349769 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.350853 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.351182 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.351609 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.352683 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.352687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.352690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.352723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.354386 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.354912 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.354917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.354919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.354999 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.355283 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.355286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.355288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.355495 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.355641 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.355946 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.356087 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5157,"thread_start_us":91,"threads_started":1}
I20260619 22:26:27.356840 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.334869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.5.176.254:41209)
0619 22:26:27.335778 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.335785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.338964 (+ 3179us) server_negotiation.cc:407] Connection header received
0619 22:26:27.339061 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.339067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.339130 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.339580 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.340848 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.348772 (+ 7924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.350023 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.350466 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.350576 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.352529 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.352532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.352535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.352556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.354444 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.354986 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.354990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.354991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.355078 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.355320 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.355324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.355326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.355503 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.355641 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.356576 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.356709 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":83,"threads_started":1}
I20260619 22:26:27.359714 11939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:27.362931 11939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:27.365559 13062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: NON_VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: true } } }
I20260619 22:26:27.365713 13062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:27.369341 13062 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: attempting to promote NON_VOTER 0dd44d2bda484738a44e935e0f8a7841 to VOTER
I20260619 22:26:27.369735 13062 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:27.371040 12733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 LEARNER]: Refusing update from remote peer f5262501ae174235a345e305a6749db8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 22:26:27.371135 12550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Refusing update from remote peer f5262501ae174235a345e305a6749db8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 22:26:27.371563 13195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 22:26:27.371711 13197 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 22:26:27.373333 13197 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.373872 12550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.374384 12733 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.375012 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.375123 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:27.375236 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.375301 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:27.375391 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.375458 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:27.376253 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.376441 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:27.376613 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5262501ae174235a345e305a6749db8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:27.376721 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:27.384734 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.379082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40835 (local address 127.0.0.1:40176)
0619 22:26:27.379259 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.379275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.379355 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.380332 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.380334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.380352 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:27.380441 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.380445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.382064 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.382068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.383224 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.383234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.384321 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.384325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.384408 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.384457 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.384510 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.384585 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 22:26:27.385391 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.379312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:34737 (local address 127.0.0.1:37046)
0619 22:26:27.380077 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.380090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.380173 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.381197 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.381200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.381218 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:27.381335 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.381342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.382633 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.382637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.383719 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.383730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.385013 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.385016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.385100 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.385121 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.385201 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.385265 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":5,"thread_start_us":115,"threads_started":1}
I20260619 22:26:27.385442 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.379243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.5.176.195:40835)
0619 22:26:27.379775 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.379803 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.379831 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:26:27.379916 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.379921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.380168 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.380345 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:27.380591 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.381935 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.383354 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.384187 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.385019 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.385146 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.385214 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.385304 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":130,"threads_started":1}
I20260619 22:26:27.385695 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.379679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.5.176.193:39413)
0619 22:26:27.380011 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.380023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.380663 (+ 640us) server_negotiation.cc:407] Connection header received
0619 22:26:27.380782 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.380787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.380973 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.381120 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:27.381663 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.382725 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.383980 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.384813 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.384927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.385054 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.385454 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.385573 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":62,"threads_started":1}
I20260619 22:26:27.385768 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.379716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.5.176.194:34737)
0619 22:26:27.380415 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.380426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.380447 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:26:27.380523 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.380528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.380914 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.381061 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:27.381464 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.382518 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.383857 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.384684 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.384788 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.384952 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.385249 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.385356 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":106,"threads_started":1}
I20260619 22:26:27.386595 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.379539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39413 (local address 127.0.0.1:48632)
0619 22:26:27.380570 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.380583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.380674 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.381232 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.381235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.381296 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:27.381537 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.381543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.382842 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.382845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.383840 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.383853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.385117 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.385120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.385213 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.385237 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.385318 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.385384 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":19,"spinlock_wait_cycles":3328,"thread_start_us":95,"threads_started":1}
I20260619 22:26:27.388473 13111 tablet_service.cc:1467] Processing CreateTablet for tablet 9d896817cc46467395a68e06bccaaf6a (DEFAULT_TABLE table=test-table [id=c30bce0f026641cba216355725170c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:27.388461 12973 tablet_service.cc:1467] Processing CreateTablet for tablet 9d896817cc46467395a68e06bccaaf6a (DEFAULT_TABLE table=test-table [id=c30bce0f026641cba216355725170c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:27.389050 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 9d896817cc46467395a68e06bccaaf6a (DEFAULT_TABLE table=test-table [id=c30bce0f026641cba216355725170c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:27.389456 12973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d896817cc46467395a68e06bccaaf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:27.389494 13111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d896817cc46467395a68e06bccaaf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:27.389894 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d896817cc46467395a68e06bccaaf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:27.396392 13211 tablet_bootstrap.cc:492] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Bootstrap starting.
I20260619 22:26:27.397796 13213 tablet_bootstrap.cc:492] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Bootstrap starting.
I20260619 22:26:27.398491 13212 tablet_bootstrap.cc:492] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Bootstrap starting.
I20260619 22:26:27.398653 13211 tablet_bootstrap.cc:654] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:27.399748 13211 log.cc:824] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:27.399893 13213 tablet_bootstrap.cc:654] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:27.400430 13212 tablet_bootstrap.cc:654] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:27.400703 13213 log.cc:824] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:27.401177 13212 log.cc:824] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:27.402264 13211 tablet_bootstrap.cc:492] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: No bootstrap required, opened a new log
I20260619 22:26:27.402473 13211 ts_tablet_manager.cc:1397] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:27.402612 13213 tablet_bootstrap.cc:492] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: No bootstrap required, opened a new log
I20260619 22:26:27.402806 13213 ts_tablet_manager.cc:1397] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:26:27.403100 13212 tablet_bootstrap.cc:492] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: No bootstrap required, opened a new log
I20260619 22:26:27.403275 13212 ts_tablet_manager.cc:1397] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:26:27.406064 13211 raft_consensus.cc:348] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.406433 13211 raft_consensus.cc:374] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:27.406490 13211 raft_consensus.cc:729] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd98a3e558f34f62aa1bde4e0f4cdf9d, State: Initialized, Role: FOLLOWER
I20260619 22:26:27.406998 13211 consensus_queue.cc:260] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.407689 13212 raft_consensus.cc:348] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.407701 13213 raft_consensus.cc:348] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.407928 13212 raft_consensus.cc:374] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:27.407928 13213 raft_consensus.cc:374] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:27.407976 13212 raft_consensus.cc:729] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7454088ed841fcb219b1e246cab7e1, State: Initialized, Role: FOLLOWER
I20260619 22:26:27.407976 13213 raft_consensus.cc:729] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97d9a6182dc4d37b1ed98f6889c6a4a, State: Initialized, Role: FOLLOWER
I20260619 22:26:27.408066 13211 ts_tablet_manager.cc:1428] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:27.408406 13212 consensus_queue.cc:260] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.408485 13040 heartbeater.cc:503] Master 127.5.176.254:41209 was elected leader, sending a full tablet report...
I20260619 22:26:27.409343 13212 ts_tablet_manager.cc:1428] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:27.409747 13178 heartbeater.cc:503] Master 127.5.176.254:41209 was elected leader, sending a full tablet report...
I20260619 22:26:27.410828 13213 consensus_queue.cc:260] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.411924 13213 ts_tablet_manager.cc:1428] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260619 22:26:27.412300 12903 heartbeater.cc:503] Master 127.5.176.254:41209 was elected leader, sending a full tablet report...
W20260619 22:26:27.579252 13180 tablet.cc:2369] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:27.615504 13042 tablet.cc:2369] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:27.648593 12905 tablet.cc:2369] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:27.677855 13219 raft_consensus.cc:482] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:27.678061 13219 raft_consensus.cc:504] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.679203 13219 leader_election.cc:288] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835), fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737)
I20260619 22:26:27.683786 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.679622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:45511 (local address 127.5.176.195:40835)
0619 22:26:27.679753 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.679757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.679939 (+ 182us) server_negotiation.cc:407] Connection header received
0619 22:26:27.680047 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.680051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.680105 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.680197 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:27.680537 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.681303 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.682687 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.683164 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.683257 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.683298 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.683524 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.683617 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:27.683871 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.679527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40835 (local address 127.5.176.193:45511)
0619 22:26:27.679781 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.679800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.679940 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.680226 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.680229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.680249 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:27.680401 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.680411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.681409 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.681413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.682563 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.682572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.683285 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.683289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.683387 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.683419 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.683500 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.683578 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":49,"thread_start_us":100,"threads_started":1}
I20260619 22:26:27.683972 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.680115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:33835 (local address 127.5.176.194:34737)
0619 22:26:27.680276 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.680281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.680392 (+ 111us) server_negotiation.cc:407] Connection header received
0619 22:26:27.680486 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.680492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.680557 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.680621 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:27.680926 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.681576 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.682894 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.683389 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.683518 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.683561 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.683765 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.683845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 22:26:27.684352 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.679966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:34737 (local address 127.5.176.193:33835)
0619 22:26:27.680296 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:27.680310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:27.680405 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:27.680652 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:27.680654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:27.680672 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:27.680781 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.680797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.681701 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.681704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.682766 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:27.682776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.683535 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.683540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:27.683635 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.683655 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 22:26:27.683731 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 22:26:27.684237 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":70,"threads_started":1}
I20260619 22:26:27.684756 13131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d896817cc46467395a68e06bccaaf6a" candidate_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c7454088ed841fcb219b1e246cab7e1" is_pre_election: true
I20260619 22:26:27.685061 13131 raft_consensus.cc:2454] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f97d9a6182dc4d37b1ed98f6889c6a4a in term 0.
I20260619 22:26:27.685063 12993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d896817cc46467395a68e06bccaaf6a" candidate_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" is_pre_election: true
I20260619 22:26:27.685339 12993 raft_consensus.cc:2454] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f97d9a6182dc4d37b1ed98f6889c6a4a in term 0.
I20260619 22:26:27.685525 12792 leader_election.cc:302] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7454088ed841fcb219b1e246cab7e1, f97d9a6182dc4d37b1ed98f6889c6a4a; no voters:
I20260619 22:26:27.685806 13219 raft_consensus.cc:2781] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:27.685899 13219 raft_consensus.cc:482] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:27.685943 13219 raft_consensus.cc:3037] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:27.686966 13219 raft_consensus.cc:504] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.687314 13219 leader_election.cc:288] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [CANDIDATE]: Term 1 election: Requested vote from peers 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835), fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737)
I20260619 22:26:27.687704 12993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d896817cc46467395a68e06bccaaf6a" candidate_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d"
I20260619 22:26:27.687667 13131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d896817cc46467395a68e06bccaaf6a" candidate_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c7454088ed841fcb219b1e246cab7e1"
I20260619 22:26:27.687820 13131 raft_consensus.cc:3037] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:27.687820 12993 raft_consensus.cc:3037] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:27.689499 13131 raft_consensus.cc:2454] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97d9a6182dc4d37b1ed98f6889c6a4a in term 1.
I20260619 22:26:27.689499 12993 raft_consensus.cc:2454] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97d9a6182dc4d37b1ed98f6889c6a4a in term 1.
I20260619 22:26:27.689793 12792 leader_election.cc:302] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7454088ed841fcb219b1e246cab7e1, f97d9a6182dc4d37b1ed98f6889c6a4a; no voters:
I20260619 22:26:27.689989 13219 raft_consensus.cc:2781] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:27.690199 13219 raft_consensus.cc:686] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 LEADER]: Becoming Leader. State: Replica: f97d9a6182dc4d37b1ed98f6889c6a4a, State: Running, Role: LEADER
I20260619 22:26:27.690575 13219 consensus_queue.cc:237] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } }
I20260619 22:26:27.694036 11937 catalog_manager.cc:5495] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a reported cstate change: term changed from 0 to 1, leader changed from <none> to f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193). New cstate: current_term: 1 leader_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } health_report { overall_health: HEALTHY } } }
I20260619 22:26:27.749964 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.746128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.5.176.193:39413)
0619 22:26:27.746312 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.746319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.746336 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:26:27.746527 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.746530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.746608 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.746730 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:27.746992 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.747526 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.748431 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.748656 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.748777 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.749104 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:27.749537 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:27.749678 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.749747 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.749806 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:26:27.755908 12993 raft_consensus.cc:1261] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Refusing update from remote peer f97d9a6182dc4d37b1ed98f6889c6a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:27.755908 13131 raft_consensus.cc:1261] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Refusing update from remote peer f97d9a6182dc4d37b1ed98f6889c6a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:27.756655 13222 consensus_queue.cc:1035] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:27.756837 13219 consensus_queue.cc:1035] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:27.765544 13227 mvcc.cc:205] Tried to move back new op lower bound from 7298695117840674816 to 7298695117583679488. Current Snapshot: MvccSnapshot[applied={T|T < 7298695117840674816}]
I20260619 22:26:27.774169 11939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:27.774358 11939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:27.778663 11939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5262501ae174235a345e305a6749db8: Sending DeleteTablet for 3 replicas of tablet 9d896817cc46467395a68e06bccaaf6a
I20260619 22:26:27.779372 13111 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d896817cc46467395a68e06bccaaf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:27 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:40176
I20260619 22:26:27.779659 12973 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d896817cc46467395a68e06bccaaf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:27 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:37046
I20260619 22:26:27.779640 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d896817cc46467395a68e06bccaaf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:27 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:48632
I20260619 22:26:27.780508 13235 tablet_replica.cc:330] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: stopping tablet replica
I20260619 22:26:27.780891 13235 raft_consensus.cc:2229] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:27.781366 13235 raft_consensus.cc:2258] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:27.781476 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 11907
I20260619 22:26:27.782929 13237 tablet_replica.cc:330] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: stopping tablet replica
I20260619 22:26:27.784054 13235 ts_tablet_manager.cc:1905] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:27.786913 13235 ts_tablet_manager.cc:1918] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:27.787016 13235 log.cc:1196] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/9d896817cc46467395a68e06bccaaf6a
I20260619 22:26:27.787349 13235 ts_tablet_manager.cc:1939] T 9d896817cc46467395a68e06bccaaf6a P 7c7454088ed841fcb219b1e246cab7e1: Deleting consensus metadata
I20260619 22:26:27.788879 13237 raft_consensus.cc:2229] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 LEADER]: Raft consensus shutting down.
W20260619 22:26:27.789083 13063 connection.cc:535] server connection from 127.0.0.1:40176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:26:27.789836 13237 raft_consensus.cc:2258] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:27.791638 13237 ts_tablet_manager.cc:1905] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:27.793063 13236 tablet_replica.cc:330] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: stopping tablet replica
I20260619 22:26:27.793503 13236 raft_consensus.cc:2229] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:27.793828 13236 raft_consensus.cc:2258] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:27.795836 13237 ts_tablet_manager.cc:1918] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:27.795933 13237 log.cc:1196] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/9d896817cc46467395a68e06bccaaf6a
I20260619 22:26:27.796267 13237 ts_tablet_manager.cc:1939] T 9d896817cc46467395a68e06bccaaf6a P f97d9a6182dc4d37b1ed98f6889c6a4a: Deleting consensus metadata
W20260619 22:26:27.797349 12791 connection.cc:441] server connection from 127.0.0.1:48632 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48632 (request call id 1) could send its response
I20260619 22:26:27.798961 13236 ts_tablet_manager.cc:1905] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:27.799612 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.781507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.5.176.253:44821)
0619 22:26:27.781869 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.781876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.786496 (+ 4620us) server_negotiation.cc:407] Connection header received
0619 22:26:27.786694 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.786700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.786765 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.786821 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.788250 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.789084 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.793045 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.793435 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.793526 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.795481 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.795488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.795494 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.795532 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.797273 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.797796 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.797802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.797804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.797912 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.798308 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.798312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.798314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.798522 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.798634 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.798983 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.799127 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260619 22:26:27.801525 13236 ts_tablet_manager.cc:1918] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:27.801617 13236 log.cc:1196] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/9d896817cc46467395a68e06bccaaf6a
I20260619 22:26:27.801951 13236 ts_tablet_manager.cc:1939] T 9d896817cc46467395a68e06bccaaf6a P fd98a3e558f34f62aa1bde4e0f4cdf9d: Deleting consensus metadata
W20260619 22:26:27.806346 12928 connection.cc:441] server connection from 127.0.0.1:37046 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37046 (request call id 1) could send its response
I20260619 22:26:27.809793 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.781107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.5.176.252:37075)
0619 22:26:27.781215 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.781220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.801377 (+ 20157us) server_negotiation.cc:407] Connection header received
0619 22:26:27.801561 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.801567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.801628 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.801682 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.802680 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.803448 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.804525 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.804890 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.804999 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.806552 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.806556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.806559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.806583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.808151 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.808606 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.808611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.808613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.808690 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.808955 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.808958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.808959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.809139 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.809237 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.809530 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.809644 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:26:27.813325 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.792901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.5.176.252:37075)
0619 22:26:27.793044 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.793048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.793715 (+ 667us) server_negotiation.cc:407] Connection header received
0619 22:26:27.793974 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.793980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.794039 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.794093 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.795580 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.796422 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.798327 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.798642 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.798717 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.800996 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.801002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.801008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.801039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.803069 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.812014 (+ 8945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.812018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.812020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.812110 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.812407 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.812410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.812412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.812611 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.812710 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.813059 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.813176 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:27.815469 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:27.802625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.5.176.253:44821)
0619 22:26:27.802730 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:27.802734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:27.806706 (+ 3972us) server_negotiation.cc:407] Connection header received
0619 22:26:27.806881 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:27.806884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:27.806929 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:27.806976 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:27.807845 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.808615 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.809800 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:27.810161 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:27.810234 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:27.812000 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:27.812004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:27.812007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:27.812033 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:27.813639 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.814113 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.814117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.814118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.814253 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:27.814528 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:27.814530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:27.814532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:27.814728 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:27.814825 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:27.815162 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 22:26:27.815285 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:26:28.771983 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:28.771522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:57333)
0619 22:26:28.771764 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:28.771873 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
W20260619 22:26:28.772332 13178 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:41209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:28.772017 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:28.771410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:46745)
0619 22:26:28.771815 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:28.771914 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":83,"threads_started":1}
W20260619 22:26:28.772737 13040 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:41209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:28.774807 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:28.774386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:52867)
0619 22:26:28.774632 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:28.774725 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
W20260619 22:26:28.775066 12903 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:41209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.279014 13248 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5262501ae174235a345e305a6749db8)
I20260619 22:26:29.279168 13248 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:29.279213 13249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5262501ae174235a345e305a6749db8)
I20260619 22:26:29.279430 13249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:29.280212 13248 leader_election.cc:288] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209), 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252:37075)
I20260619 22:26:29.280536 13249 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209), e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253:44821)
I20260619 22:26:29.280761 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.280080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:37942)
0619 22:26:29.280569 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.280660 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":40,"threads_started":1}
W20260619 22:26:29.280930 12527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.281210 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.280789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:37952)
0619 22:26:29.281054 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.281120 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":52,"threads_started":1}
W20260619 22:26:29.281354 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.282285 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.282000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:37966)
0619 22:26:29.282125 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.282217 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 22:26:29.282440 12527 leader_election.cc:334] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.283263 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.283050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:37972)
0619 22:26:29.283157 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.283202 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:26:29.283427 12710 leader_election.cc:334] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.285766 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.280027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:37075 (local address 127.0.0.1:50058)
0619 22:26:29.280505 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.280520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:29.281594 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:29.281940 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:29.281943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:29.281957 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:29.282049 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:29.282058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.283175 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.283179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:29.284447 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:29.284458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.285317 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.285321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:29.285429 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:29.285454 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:26:29.285540 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 22:26:29.285635 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260619 22:26:29.285766 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.281003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44821 (local address 127.0.0.1:53548)
0619 22:26:29.281281 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.281295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:29.281387 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:29.281853 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:29.281858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:29.281887 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:29.282009 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:29.282015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.283175 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.283179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:29.284447 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:29.284458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.285317 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.285321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:29.285437 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:29.285457 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 22:26:29.285540 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 22:26:29.285633 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260619 22:26:29.285795 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.281223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.5.176.253:44821)
0619 22:26:29.281599 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:29.281608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:29.281622 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:29.281669 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:29.281673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:29.281733 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:29.282107 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:29.282222 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.283049 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.284595 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.285187 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.285291 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:29.285352 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:29.285567 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 22:26:29.285673 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":303,"thread_start_us":91,"threads_started":1}
I20260619 22:26:29.285795 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.280771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.5.176.252:37075)
0619 22:26:29.281717 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:29.281721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:29.281733 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:29.281773 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:29.281778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:29.281829 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:29.282169 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:29.282251 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.283049 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.284595 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:29.285187 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:29.285291 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:29.285352 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:29.285567 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 22:26:29.285673 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":891,"thread_start_us":61,"threads_started":1}
I20260619 22:26:29.286420 12733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0dd44d2bda484738a44e935e0f8a7841" is_pre_election: true
I20260619 22:26:29.286548 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dd44d2bda484738a44e935e0f8a7841" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" is_pre_election: true
I20260619 22:26:29.286639 12733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56a7b2ea9224e5dbfaf58dad68b0c96 in term 1.
I20260619 22:26:29.286736 12550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dd44d2bda484738a44e935e0f8a7841 in term 1.
I20260619 22:26:29.286952 12526 leader_election.cc:302] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dd44d2bda484738a44e935e0f8a7841, e56a7b2ea9224e5dbfaf58dad68b0c96; no voters: f5262501ae174235a345e305a6749db8
I20260619 22:26:29.287063 12708 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dd44d2bda484738a44e935e0f8a7841, e56a7b2ea9224e5dbfaf58dad68b0c96; no voters: f5262501ae174235a345e305a6749db8
I20260619 22:26:29.287182 13248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 22:26:29.287240 13248 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5262501ae174235a345e305a6749db8)
I20260619 22:26:29.287285 13248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:26:29.287272 13249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 22:26:29.287330 13249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5262501ae174235a345e305a6749db8)
I20260619 22:26:29.287379 13249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:26:29.288290 13248 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:29.288301 13249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:29.288672 13248 leader_election.cc:288] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 election: Requested vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209), 0dd44d2bda484738a44e935e0f8a7841 (127.5.176.252:37075)
I20260619 22:26:29.288677 13249 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 2 election: Requested vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209), e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253:44821)
I20260619 22:26:29.289004 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.288739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:37980)
0619 22:26:29.288897 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.288947 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 22:26:29.289004 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.288745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:37994)
0619 22:26:29.288905 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.288947 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 22:26:29.289209 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dd44d2bda484738a44e935e0f8a7841" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
I20260619 22:26:29.289384 12550 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0dd44d2bda484738a44e935e0f8a7841 in current term 2: Already voted for candidate e56a7b2ea9224e5dbfaf58dad68b0c96 in this term.
I20260619 22:26:29.289630 12733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0dd44d2bda484738a44e935e0f8a7841"
I20260619 22:26:29.289709 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.289440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38004)
0619 22:26:29.289601 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.289654 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 22:26:29.289780 12733 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e56a7b2ea9224e5dbfaf58dad68b0c96 in current term 2: Already voted for candidate 0dd44d2bda484738a44e935e0f8a7841 in this term.
W20260619 22:26:29.289793 12710 leader_election.cc:334] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.289875 12710 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0dd44d2bda484738a44e935e0f8a7841; no voters: e56a7b2ea9224e5dbfaf58dad68b0c96, f5262501ae174235a345e305a6749db8
I20260619 22:26:29.290012 13249 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 22:26:29.290446 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.290236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38008)
0619 22:26:29.290338 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.290386 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 22:26:29.290542 12527 leader_election.cc:334] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:29.290604 12527 leader_election.cc:302] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e56a7b2ea9224e5dbfaf58dad68b0c96; no voters: 0dd44d2bda484738a44e935e0f8a7841, f5262501ae174235a345e305a6749db8
I20260619 22:26:29.290727 13248 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 22:26:29.773450 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.772919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:51477)
0619 22:26:29.773268 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.773338 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
I20260619 22:26:29.773645 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.773163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:57925)
0619 22:26:29.773431 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.773491 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":64,"threads_started":1}
I20260619 22:26:29.775938 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:29.775504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:52401)
0619 22:26:29.775795 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:29.775857 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":64,"threads_started":1}
I20260619 22:26:30.774708 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.774249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:37281)
0619 22:26:30.774522 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.774596 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260619 22:26:30.774952 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.774249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:48953)
0619 22:26:30.774704 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.774766 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":79,"threads_started":1}
I20260619 22:26:30.776991 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.776636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:46395)
0619 22:26:30.776861 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.776914 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260619 22:26:30.890522 13264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:30.890740 13264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:30.891218 13264 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209), e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253:44821)
I20260619 22:26:30.891858 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.891447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38282)
0619 22:26:30.891688 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.891755 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260619 22:26:30.891993 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dd44d2bda484738a44e935e0f8a7841" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" is_pre_election: true
I20260619 22:26:30.892163 12550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dd44d2bda484738a44e935e0f8a7841 in term 2.
I20260619 22:26:30.892478 12708 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd44d2bda484738a44e935e0f8a7841, e56a7b2ea9224e5dbfaf58dad68b0c96; no voters:
I20260619 22:26:30.892640 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.892435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38292)
0619 22:26:30.892525 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.892572 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 22:26:30.892659 13264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 22:26:30.892738 13264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260619 22:26:30.892753 12710 leader_election.cc:334] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:30.892795 13264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:26:30.893811 13264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:30.894171 13264 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 3 election: Requested vote from peers f5262501ae174235a345e305a6749db8 (127.5.176.254:41209), e56a7b2ea9224e5dbfaf58dad68b0c96 (127.5.176.253:44821)
I20260619 22:26:30.894461 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.894248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38300)
0619 22:26:30.894327 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.894375 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 22:26:30.894526 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dd44d2bda484738a44e935e0f8a7841" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96"
I20260619 22:26:30.894672 12550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:26:30.895104 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.894901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38304)
0619 22:26:30.895005 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.895053 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:26:30.895215 12710 leader_election.cc:334] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
I20260619 22:26:30.895780 12550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dd44d2bda484738a44e935e0f8a7841 in term 3.
I20260619 22:26:30.896080 12708 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dd44d2bda484738a44e935e0f8a7841, e56a7b2ea9224e5dbfaf58dad68b0c96; no voters: f5262501ae174235a345e305a6749db8
I20260619 22:26:30.896229 13264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 22:26:30.896451 13264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [term 3 LEADER]: Becoming Leader. State: Replica: 0dd44d2bda484738a44e935e0f8a7841, State: Running, Role: LEADER
I20260619 22:26:30.896721 13264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } }
I20260619 22:26:30.898039 13267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd44d2bda484738a44e935e0f8a7841. Latest consensus state: current_term: 3 leader_uuid: "0dd44d2bda484738a44e935e0f8a7841" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:30.898180 13267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:30.898483 13269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:30.900628 13269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:30.900967 13269 catalog_manager.cc:1259] Loaded cluster ID: f1068e962853462ba604b42122c242c4
I20260619 22:26:30.901010 13269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:30.901491 13269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:30.901824 13269 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Loaded TSK: 0
I20260619 22:26:30.902544 13269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:30.950227 12723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:30.954070 12723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:30.957237 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.956937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38322)
0619 22:26:30.957083 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.957149 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 22:26:30.957267 12550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [term 3 FOLLOWER]: Refusing update from remote peer 0dd44d2bda484738a44e935e0f8a7841: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 22:26:30.957767 13264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 22:26:30.957996 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.957733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.0.0.1:38336)
0619 22:26:30.957837 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.957917 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 22:26:30.958102 12710 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 -> Peer f5262501ae174235a345e305a6749db8 (127.5.176.254:41209): Couldn't send request to peer f5262501ae174235a345e305a6749db8. Status: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 22:26:30.959311 13271 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd44d2bda484738a44e935e0f8a7841. Latest consensus state: current_term: 3 leader_uuid: "0dd44d2bda484738a44e935e0f8a7841" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:30.959442 13271 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:30.960595 13264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0dd44d2bda484738a44e935e0f8a7841" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:30.960711 13264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:30.961170 13271 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0dd44d2bda484738a44e935e0f8a7841" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5262501ae174235a345e305a6749db8" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 41209 } } peers { permanent_uuid: "e56a7b2ea9224e5dbfaf58dad68b0c96" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "0dd44d2bda484738a44e935e0f8a7841" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 37075 } attrs { promote: false } } }
I20260619 22:26:30.961295 13271 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56a7b2ea9224e5dbfaf58dad68b0c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:26:30.971958 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.967195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:34737 (local address 127.0.0.1:55304)
0619 22:26:30.967291 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.967317 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:30.967425 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:30.967945 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:30.967948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:30.967966 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:30.968079 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:30.968086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.969203 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.969206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:30.970732 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:30.970742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.971598 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.971602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:30.971691 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:30.971713 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:26:30.971788 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:26:30.971844 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":174}
I20260619 22:26:30.972044 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.967296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.5.176.194:34737)
0619 22:26:30.967660 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:30.967671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:30.967689 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:26:30.967745 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:30.967749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:30.967813 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:30.968198 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:30.968322 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.969080 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.970865 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.971469 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.971571 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:30.971639 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:30.971831 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 22:26:30.971907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":90,"threads_started":1}
I20260619 22:26:30.972551 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.967763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39413 (local address 127.0.0.1:47158)
0619 22:26:30.968036 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.968051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:30.968141 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:30.968561 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:30.968564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:30.968582 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:30.968678 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:30.968684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.969799 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.969802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:30.970939 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:30.970946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.972148 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.972153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:30.972262 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:30.972291 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:26:30.972352 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 22:26:30.972425 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260619 22:26:30.972721 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.968417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40835 (local address 127.0.0.1:36688)
0619 22:26:30.968656 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.968670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:30.968746 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:30.969191 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:30.969194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:30.969212 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:30.969359 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:30.969364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.970382 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.970387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:30.971398 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:30.971408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.972293 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.972306 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:30.972396 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:30.972420 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:26:30.972472 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 22:26:30.972526 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":37,"thread_start_us":82,"threads_started":1}
I20260619 22:26:30.972779 12973 tablet_service.cc:1467] Processing CreateTablet for tablet 5e76eb697aca49a6bd6e279378350848 (DEFAULT_TABLE table=test-table [id=8876942c7a694dc19272a78bfc6b94de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:30.972968 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.967879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.5.176.193:39413)
0619 22:26:30.968156 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:30.968162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:30.968181 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:26:30.968246 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:30.968249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:30.968318 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:30.968411 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:30.968804 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.969546 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.971046 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.971472 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.971604 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:30.971666 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:30.972751 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0619 22:26:30.972845 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260619 22:26:30.972970 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.968744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.5.176.195:40835)
0619 22:26:30.968962 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:30.968966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:30.968978 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:30.969024 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:30.969026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:30.969079 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:30.969163 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:30.969504 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.970252 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.971552 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:30.972159 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:30.972639 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:30.972700 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:30.972736 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 22:26:30.972818 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260619 22:26:30.973086 12973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e76eb697aca49a6bd6e279378350848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:30.973443 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 5e76eb697aca49a6bd6e279378350848 (DEFAULT_TABLE table=test-table [id=8876942c7a694dc19272a78bfc6b94de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:30.973708 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e76eb697aca49a6bd6e279378350848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:30.975415 13281 tablet_bootstrap.cc:492] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Bootstrap starting.
I20260619 22:26:30.976024 13282 tablet_bootstrap.cc:492] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a: Bootstrap starting.
I20260619 22:26:30.976197 13111 tablet_service.cc:1467] Processing CreateTablet for tablet 5e76eb697aca49a6bd6e279378350848 (DEFAULT_TABLE table=test-table [id=8876942c7a694dc19272a78bfc6b94de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:30.976336 13281 tablet_bootstrap.cc:654] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:30.976490 13111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e76eb697aca49a6bd6e279378350848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:30.976888 13282 tablet_bootstrap.cc:654] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:30.977962 13281 tablet_bootstrap.cc:492] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: No bootstrap required, opened a new log
I20260619 22:26:30.978057 13281 ts_tablet_manager.cc:1397] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:30.978093 13282 tablet_bootstrap.cc:492] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a: No bootstrap required, opened a new log
I20260619 22:26:30.978204 13282 ts_tablet_manager.cc:1397] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 22:26:30.978571 13282 raft_consensus.cc:348] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:30.978552 13281 raft_consensus.cc:348] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:30.978696 13281 raft_consensus.cc:374] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:30.978695 13282 raft_consensus.cc:374] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:30.978740 13282 raft_consensus.cc:729] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97d9a6182dc4d37b1ed98f6889c6a4a, State: Initialized, Role: FOLLOWER
I20260619 22:26:30.978740 13281 raft_consensus.cc:729] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd98a3e558f34f62aa1bde4e0f4cdf9d, State: Initialized, Role: FOLLOWER
I20260619 22:26:30.978878 13281 consensus_queue.cc:260] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:30.978924 13282 consensus_queue.cc:260] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:30.979269 13281 ts_tablet_manager.cc:1428] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 22:26:30.979300 13282 ts_tablet_manager.cc:1428] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:30.980484 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.980178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:53299)
0619 22:26:30.980347 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.980413 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 22:26:30.980672 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.979949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:47433)
0619 22:26:30.980548 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.980603 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
I20260619 22:26:30.982026 13285 tablet_bootstrap.cc:492] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Bootstrap starting.
I20260619 22:26:30.982576 13039 heartbeater.cc:503] Master 127.5.176.252:37075 was elected leader, sending a full tablet report...
I20260619 22:26:30.982743 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.982051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:39949)
0619 22:26:30.982622 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.982676 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499}
I20260619 22:26:30.983134 13285 tablet_bootstrap.cc:654] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:30.983496 12902 heartbeater.cc:503] Master 127.5.176.252:37075 was elected leader, sending a full tablet report...
I20260619 22:26:30.984457 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.984207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:57211)
0619 22:26:30.984327 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.984375 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 22:26:30.985132 13285 tablet_bootstrap.cc:492] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: No bootstrap required, opened a new log
I20260619 22:26:30.985244 13285 ts_tablet_manager.cc:1397] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:30.985636 13285 raft_consensus.cc:348] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:30.985754 13285 raft_consensus.cc:374] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:30.985797 13285 raft_consensus.cc:729] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7454088ed841fcb219b1e246cab7e1, State: Initialized, Role: FOLLOWER
I20260619 22:26:30.985944 13285 consensus_queue.cc:260] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:30.986361 13285 ts_tablet_manager.cc:1428] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:30.987088 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.986835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:38743)
0619 22:26:30.986981 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.987033 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 22:26:30.987742 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:30.987516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:52989)
0619 22:26:30.987628 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:30.987681 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:26:30.988150 13177 heartbeater.cc:503] Master 127.5.176.252:37075 was elected leader, sending a full tablet report...
I20260619 22:26:31.117597 13286 raft_consensus.cc:482] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:31.117746 13286 raft_consensus.cc:504] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:31.118965 13286 leader_election.cc:288] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413), 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835)
I20260619 22:26:31.123128 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.119447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:55945 (local address 127.5.176.195:40835)
0619 22:26:31.119609 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:31.119618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:31.119630 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:31.119674 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:31.119676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:31.119715 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:31.119810 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:31.120088 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.120677 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.121923 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.122512 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.122645 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:31.122704 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:31.122926 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 22:26:31.122997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:26:31.123128 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.119178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40835 (local address 127.5.176.194:55945)
0619 22:26:31.119323 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.119342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:31.119466 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:31.119821 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:31.119824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:31.119843 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:31.119971 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:31.119977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.120788 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.120793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:31.121792 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:31.121801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.122652 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.122656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:31.122794 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:31.122816 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:26:31.122907 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 22:26:31.122990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":13}
I20260619 22:26:31.123137 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.119365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:55875 (local address 127.5.176.193:39413)
0619 22:26:31.119493 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:31.119499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:31.119930 (+ 431us) server_negotiation.cc:407] Connection header received
0619 22:26:31.120080 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:31.120083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:31.120134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:31.120214 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:31.120517 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.121015 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.122069 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.122503 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.122645 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:31.122704 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:31.122946 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 22:26:31.123018 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:26:31.123152 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.119178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39413 (local address 127.5.176.194:55875)
0619 22:26:31.119833 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.119846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:31.119967 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:31.120237 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:31.120240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:31.120255 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:31.120392 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:31.120398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.121128 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.121134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:31.121951 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:31.121959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.122647 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.122651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:31.122824 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:31.122841 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:26:31.122931 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 22:26:31.123003 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":380,"thread_start_us":134,"threads_started":1}
I20260619 22:26:31.123711 13131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e76eb697aca49a6bd6e279378350848" candidate_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c7454088ed841fcb219b1e246cab7e1" is_pre_election: true
I20260619 22:26:31.123880 13131 raft_consensus.cc:2454] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd98a3e558f34f62aa1bde4e0f4cdf9d in term 0.
I20260619 22:26:31.123845 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e76eb697aca49a6bd6e279378350848" candidate_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" is_pre_election: true
I20260619 22:26:31.124068 12856 raft_consensus.cc:2454] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd98a3e558f34f62aa1bde4e0f4cdf9d in term 0.
I20260619 22:26:31.124277 12929 leader_election.cc:302] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7454088ed841fcb219b1e246cab7e1, fd98a3e558f34f62aa1bde4e0f4cdf9d; no voters:
I20260619 22:26:31.124624 13286 raft_consensus.cc:2781] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:31.124687 13286 raft_consensus.cc:482] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:31.124715 13286 raft_consensus.cc:3037] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:31.125803 13286 raft_consensus.cc:504] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:31.126240 13286 leader_election.cc:288] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [CANDIDATE]: Term 1 election: Requested vote from peers f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413), 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835)
I20260619 22:26:31.126513 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e76eb697aca49a6bd6e279378350848" candidate_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a"
I20260619 22:26:31.126591 13131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e76eb697aca49a6bd6e279378350848" candidate_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c7454088ed841fcb219b1e246cab7e1"
I20260619 22:26:31.126662 12856 raft_consensus.cc:3037] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:31.126744 13131 raft_consensus.cc:3037] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:31.127751 13131 raft_consensus.cc:2454] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd98a3e558f34f62aa1bde4e0f4cdf9d in term 1.
I20260619 22:26:31.127776 12856 raft_consensus.cc:2454] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd98a3e558f34f62aa1bde4e0f4cdf9d in term 1.
I20260619 22:26:31.128046 12929 leader_election.cc:302] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7454088ed841fcb219b1e246cab7e1, fd98a3e558f34f62aa1bde4e0f4cdf9d; no voters:
I20260619 22:26:31.128225 13286 raft_consensus.cc:2781] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:31.128500 13286 raft_consensus.cc:686] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 LEADER]: Becoming Leader. State: Replica: fd98a3e558f34f62aa1bde4e0f4cdf9d, State: Running, Role: LEADER
I20260619 22:26:31.128804 13286 consensus_queue.cc:237] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } }
I20260619 22:26:31.129231 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.128992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:33241)
0619 22:26:31.129087 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.129143 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 22:26:31.131505 12722 catalog_manager.cc:5495] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194). New cstate: current_term: 1 leader_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd98a3e558f34f62aa1bde4e0f4cdf9d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:31.162943 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.159500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.5.176.194:34737)
0619 22:26:31.159626 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:31.159630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:31.159643 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:31.159715 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:31.159717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:31.159769 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:31.159881 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:31.160092 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.160586 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.161534 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:31.161743 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:31.161826 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:31.162053 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:31.162586 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:31.162676 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:31.162757 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 22:26:31.162831 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:26:31.165724 13131 raft_consensus.cc:1261] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Refusing update from remote peer fd98a3e558f34f62aa1bde4e0f4cdf9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:31.166074 12856 raft_consensus.cc:1261] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Refusing update from remote peer fd98a3e558f34f62aa1bde4e0f4cdf9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:31.166306 13291 consensus_queue.cc:1035] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c7454088ed841fcb219b1e246cab7e1" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:31.166703 13286 consensus_queue.cc:1035] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97d9a6182dc4d37b1ed98f6889c6a4a" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:31.168083 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.167780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:47769)
0619 22:26:31.167927 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.167998 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 22:26:31.169749 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.169500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:53465)
0619 22:26:31.169608 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.169671 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 22:26:31.170454 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.170195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:58823)
0619 22:26:31.170330 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.170383 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 22:26:31.171021 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.170733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.195:50543)
0619 22:26:31.170887 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.170952 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 22:26:31.171747 13294 mvcc.cc:205] Tried to move back new op lower bound from 7298695131809894400 to 7298695131666415616. Current Snapshot: MvccSnapshot[applied={T|T < 7298695131809894400 or (T in {7298695131809894400})}]
I20260619 22:26:31.173892 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.173616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.194:38031)
0619 22:26:31.173775 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.173824 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 22:26:31.176772 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:31.176469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:41209 (local address 127.5.176.193:55143)
0619 22:26:31.176631 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.176698 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 22:26:31.178572 12722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:31.178726 12722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:31.183451 12722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dd44d2bda484738a44e935e0f8a7841: Sending DeleteTablet for 3 replicas of tablet 5e76eb697aca49a6bd6e279378350848
I20260619 22:26:31.184147 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e76eb697aca49a6bd6e279378350848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:31 UTC) from {username='oryx', principal='oryx/127.5.176.252@KRBTEST.COM'} at 127.0.0.1:47158
I20260619 22:26:31.184379 12973 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e76eb697aca49a6bd6e279378350848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:31 UTC) from {username='oryx', principal='oryx/127.5.176.252@KRBTEST.COM'} at 127.0.0.1:55304
I20260619 22:26:31.184518 13303 tablet_replica.cc:330] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a: stopping tablet replica
I20260619 22:26:31.184641 13303 raft_consensus.cc:2229] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:31.184721 13111 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e76eb697aca49a6bd6e279378350848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:31 UTC) from {username='oryx', principal='oryx/127.5.176.252@KRBTEST.COM'} at 127.0.0.1:36688
I20260619 22:26:31.184780 13303 raft_consensus.cc:2258] T 5e76eb697aca49a6bd6e279378350848 P f97d9a6182dc4d37b1ed98f6889c6a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:31.184995 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12684
I20260619 22:26:31.185143 13305 tablet_replica.cc:330] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: stopping tablet replica
I20260619 22:26:31.185283 13305 raft_consensus.cc:2229] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:31.185415 13305 raft_consensus.cc:2258] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:31.186264 13305 ts_tablet_manager.cc:1905] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:31.187258 13304 tablet_replica.cc:330] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: stopping tablet replica
I20260619 22:26:31.187427 13304 raft_consensus.cc:2229] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:31.187827 13304 raft_consensus.cc:2258] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:31.188555 13304 ts_tablet_manager.cc:1905] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:31.189510 13305 ts_tablet_manager.cc:1918] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:31.189599 13305 log.cc:1196] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/5e76eb697aca49a6bd6e279378350848
I20260619 22:26:31.189978 13305 ts_tablet_manager.cc:1939] T 5e76eb697aca49a6bd6e279378350848 P 7c7454088ed841fcb219b1e246cab7e1: Deleting consensus metadata
I20260619 22:26:31.190802 12710 catalog_manager.cc:4841] TS 7c7454088ed841fcb219b1e246cab7e1 (127.5.176.195:40835): tablet 5e76eb697aca49a6bd6e279378350848 (table test-table [id=8876942c7a694dc19272a78bfc6b94de]) successfully deleted
I20260619 22:26:31.191735 13304 ts_tablet_manager.cc:1918] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:31.191816 13304 log.cc:1196] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/5e76eb697aca49a6bd6e279378350848
I20260619 22:26:31.192088 13304 ts_tablet_manager.cc:1939] T 5e76eb697aca49a6bd6e279378350848 P fd98a3e558f34f62aa1bde4e0f4cdf9d: Deleting consensus metadata
I20260619 22:26:31.192807 12710 catalog_manager.cc:4841] TS fd98a3e558f34f62aa1bde4e0f4cdf9d (127.5.176.194:34737): tablet 5e76eb697aca49a6bd6e279378350848 (table test-table [id=8876942c7a694dc19272a78bfc6b94de]) successfully deleted
W20260619 22:26:31.195287 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:31.195014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39413 (local address 127.0.0.1:47174)
0619 22:26:31.195146 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:31.195191 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:39413: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 22:26:31.195542 12709 catalog_manager.cc:4584] TS f97d9a6182dc4d37b1ed98f6889c6a4a (127.5.176.193:39413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e76eb697aca49a6bd6e279378350848: Network error: Client connection negotiation failed: client connection to 127.5.176.193:39413: connect: Connection reset by peer (error 104)
I20260619 22:26:31.195727 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12910
I20260619 22:26:31.202536 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 13046
I20260619 22:26:31.208401 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12403
I20260619 22:26:31.214284 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 12512
2026-06-19T22:26:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7308 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:31 dist-test-slave-bt27 krb5kdc[13308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:31 dist-test-slave-bt27 krb5kdc[13308](info): set up 2 sockets
Jun 19 22:26:31 dist-test-slave-bt27 krb5kdc[13308](info): commencing operation
krb5kdc: starting...
W20260619 22:26:32.274912 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:32 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab.
Entry for principal oryx/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab.
I20260619 22:26:32.402428 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:33073
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab
--principal=oryx/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:32.516359 13321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:32.516633 13321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:32.516712 13321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:32.520902 13321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:32.520977 13321 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 22:26:32.520994 13321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:32.521015 13321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:32.521031 13321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:32.521073 13321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:32.526638 13321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:33073
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.254.keytab
--principal=oryx/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:32.527813 13321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:32.528918 13321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:32.537433 13329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:32.537432 13327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:32.537506 13321 server_base.cc:1034] running on GCE node
W20260619 22:26:32.537585 13326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:32.538290 13321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:32.539386 13321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:32.540545 13321 hybrid_clock.cc:648] HybridClock initialized: now 1781907992540534 us; error 27 us; skew 500 ppm
Jun 19 22:26:32 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907992, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:32.543388 13321 init.cc:377] Logged in from keytab as oryx/127.5.176.254@KRBTEST.COM (short username oryx)
I20260619 22:26:32.544284 13321 webserver.cc:460] Webserver started at http://127.5.176.254:39753/ using document root <none> and password file <none>
I20260619 22:26:32.544898 13321 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:32.544960 13321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:32.545132 13321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:32.547204 13321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "d8b66f544d7547d3a99984f2ec80bec3"
format_stamp: "Formatted at 2026-06-19 22:26:32 on dist-test-slave-bt27"
server_key: "d9cb2b15ba15246bda01ac77f0279c68"
server_key_iv: "af0aab63cb82e3a630035371fb2506e7"
server_key_version: "encryptionkey@0"
I20260619 22:26:32.547706 13321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "d8b66f544d7547d3a99984f2ec80bec3"
format_stamp: "Formatted at 2026-06-19 22:26:32 on dist-test-slave-bt27"
server_key: "d9cb2b15ba15246bda01ac77f0279c68"
server_key_iv: "af0aab63cb82e3a630035371fb2506e7"
server_key_version: "encryptionkey@0"
I20260619 22:26:32.551365 13321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:26:32.554073 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:32.555580 13321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:32.555716 13321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "d8b66f544d7547d3a99984f2ec80bec3"
format_stamp: "Formatted at 2026-06-19 22:26:32 on dist-test-slave-bt27"
server_key: "d9cb2b15ba15246bda01ac77f0279c68"
server_key_iv: "af0aab63cb82e3a630035371fb2506e7"
server_key_version: "encryptionkey@0"
I20260619 22:26:32.555843 13321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:32.567673 13321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:32.570933 13321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:32.571141 13321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:32.579737 13321 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:33073
I20260619 22:26:32.579751 13398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:33073 every 8 connection(s)
I20260619 22:26:32.580755 13321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:32.583679 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:32.589507 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 13321
I20260619 22:26:32.589505 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Bootstrap starting.
I20260619 22:26:32.589628 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:32.589918 5827 external_mini_cluster.cc:1428] Setting key f3e1013f903f0e41f02b865dda0db642
I20260619 22:26:32.592257 13399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:32.593012 13399 log.cc:824] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:32.595036 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: No bootstrap required, opened a new log
I20260619 22:26:32.598608 13399 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b66f544d7547d3a99984f2ec80bec3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33073 } }
I20260619 22:26:32.598852 13399 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 22:26:32 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:32.598917 13399 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b66f544d7547d3a99984f2ec80bec3, State: Initialized, Role: FOLLOWER
I20260619 22:26:32.599393 13399 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b66f544d7547d3a99984f2ec80bec3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33073 } }
I20260619 22:26:32.599520 13399 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:32.599607 13399 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:32.599705 13399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:32.600620 13399 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b66f544d7547d3a99984f2ec80bec3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33073 } }
I20260619 22:26:32.600980 13399 leader_election.cc:302] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8b66f544d7547d3a99984f2ec80bec3; no voters:
I20260619 22:26:32.601330 13399 leader_election.cc:288] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:32.601411 13404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:32.601581 13404 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [term 1 LEADER]: Becoming Leader. State: Replica: d8b66f544d7547d3a99984f2ec80bec3, State: Running, Role: LEADER
I20260619 22:26:32.601881 13404 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b66f544d7547d3a99984f2ec80bec3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33073 } }
I20260619 22:26:32.602686 13399 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:32.603114 13405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8b66f544d7547d3a99984f2ec80bec3. Latest consensus state: current_term: 1 leader_uuid: "d8b66f544d7547d3a99984f2ec80bec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b66f544d7547d3a99984f2ec80bec3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33073 } } }
I20260619 22:26:32.603248 13405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:32.603943 13406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8b66f544d7547d3a99984f2ec80bec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b66f544d7547d3a99984f2ec80bec3" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33073 } } }
I20260619 22:26:32.604039 13406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:32.605351 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:32.591360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.5.176.254:33073)
0619 22:26:32.591866 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:32.591882 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:32.591918 (+ 36us) server_negotiation.cc:407] Connection header received
0619 22:26:32.592684 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:32.592704 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:32.593042 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:32.593446 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:32.594318 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.595359 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.596224 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.596698 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.596868 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:32.599456 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:32.599462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:32.599476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:32.599527 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:32.601492 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:32.602032 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:32.602041 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:32.602047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:32.602213 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:32.602529 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:32.602532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:32.602534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:32.603054 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:32.603268 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:32.603577 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 22:26:32.603964 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":171,"threads_started":1}
W20260619 22:26:32.607637 13419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:26:32.607717 13419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:26:32.607779 13408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:32.608458 13408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:32.614173 13408 catalog_manager.cc:1347] Generated new cluster ID: d0097e3c33f14ba8919a3d479572a214
I20260619 22:26:32.614253 13408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:32.628921 13408 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:32.630313 13408 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:32.635135 13408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Generated new TSK 0
I20260619 22:26:32.635957 13408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab.
Entry for principal oryx/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab.
I20260619 22:26:32.669821 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33073
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:32.786003 13425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:32.786298 13425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:32.786391 13425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:32.790535 13425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:32.790617 13425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:32.790659 13425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:32.790695 13425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:32.796314 13425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:32.797454 13425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:32.798537 13425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:32.808571 13433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:32.808547 13431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:32.808527 13430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:32.808771 13425 server_base.cc:1034] running on GCE node
I20260619 22:26:32.809266 13425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:32.809970 13425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:32.811136 13425 hybrid_clock.cc:648] HybridClock initialized: now 1781907992811100 us; error 52 us; skew 500 ppm
Jun 19 22:26:32 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907992, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:32.814391 13425 init.cc:377] Logged in from keytab as oryx/127.5.176.193@KRBTEST.COM (short username oryx)
I20260619 22:26:32.815367 13425 webserver.cc:460] Webserver started at http://127.5.176.193:37381/ using document root <none> and password file <none>
I20260619 22:26:32.815973 13425 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:32.816069 13425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:32.816308 13425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:32.818511 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "e79be34f85714ea19d499cfe703ad1fb"
format_stamp: "Formatted at 2026-06-19 22:26:32 on dist-test-slave-bt27"
server_key: "d1fd3a638781bc23eefae178d11e3ef2"
server_key_iv: "3bc240029eb390887f90fc22eeed10ee"
server_key_version: "encryptionkey@0"
I20260619 22:26:32.819093 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "e79be34f85714ea19d499cfe703ad1fb"
format_stamp: "Formatted at 2026-06-19 22:26:32 on dist-test-slave-bt27"
server_key: "d1fd3a638781bc23eefae178d11e3ef2"
server_key_iv: "3bc240029eb390887f90fc22eeed10ee"
server_key_version: "encryptionkey@0"
I20260619 22:26:32.822962 13425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:26:32.825626 13440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:32.826988 13425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:32.827136 13425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "e79be34f85714ea19d499cfe703ad1fb"
format_stamp: "Formatted at 2026-06-19 22:26:32 on dist-test-slave-bt27"
server_key: "d1fd3a638781bc23eefae178d11e3ef2"
server_key_iv: "3bc240029eb390887f90fc22eeed10ee"
server_key_version: "encryptionkey@0"
I20260619 22:26:32.827263 13425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:32.854996 13425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:32.858871 13425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:32.859162 13425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:32.861019 13425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:32.861099 13425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:32.861178 13425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:32.861219 13425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:32 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907992, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:32.873063 13555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:36023 every 8 connection(s)
I20260619 22:26:32.872937 13425 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:36023
I20260619 22:26:32.874634 13425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:32.877234 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 13425
I20260619 22:26:32.877344 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:32.877712 5827 external_mini_cluster.cc:1428] Setting key fbd71049adab9609c4d0cb52fb3414d8
I20260619 22:26:32.877862 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:32.861680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47033 (local address 127.5.176.254:33073)
0619 22:26:32.861824 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:32.861831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:32.863368 (+ 1537us) server_negotiation.cc:407] Connection header received
0619 22:26:32.864916 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:32.864919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:32.864972 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:32.865041 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:32.867046 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.867704 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.869729 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.869973 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.870073 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:32.873655 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:32.873659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:32.873662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:32.873693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:32.875430 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:32.876124 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:32.876127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:32.876129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:32.876211 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:32.876607 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:32.876611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:32.876612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:32.876845 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:32.876935 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:32.877532 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 22:26:32.877681 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:26:32.878808 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:32.862634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.5.176.193:47033)
0619 22:26:32.863173 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:32.863245 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:32.864651 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:32.865312 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:32.865328 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:32.865950 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:32.866854 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:32.866883 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.867844 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.867848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:32.869546 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:32.869560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.870196 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.870202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:32.870452 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:32.871236 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:32.871269 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:32.873411 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:32.875584 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:32.875590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:32.875606 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:32.876029 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:32.876323 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:32.876325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:32.876327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:32.876467 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:32.876947 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:32.876952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:32.877184 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 22:26:32.877388 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 22:26:32.877720 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260619 22:26:32.880056 13556 heartbeater.cc:347] Connected to a master server at 127.5.176.254:33073
I20260619 22:26:32.880232 13556 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:32.880733 13556 heartbeater.cc:511] Master 127.5.176.254:33073 requested a full tablet report, sending...
I20260619 22:26:32.882504 13351 ts_manager.cc:194] Registered new tserver with Master: e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023)
I20260619 22:26:32.884140 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:47033
WARNING: no policy specified for oryx/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:32.891641 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:32.884125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55917 (local address 127.5.176.254:33073)
0619 22:26:32.884276 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:32.884282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:32.884298 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:32.884334 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:32.884339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:32.884390 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:32.884505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:32.885008 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.885916 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.886856 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.887075 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.887187 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:32.888340 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:32.888344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:32.888347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:32.888375 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:32.889908 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:32.890418 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:32.890421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:32.890423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:32.890503 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:32.890795 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:32.890798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:32.890800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:32.890978 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:32.891085 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:32.891361 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 22:26:32.891470 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:26:32.891656 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:32.882411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.5.176.193:55917)
0619 22:26:32.882693 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:32.882705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:32.882786 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:32.884520 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:32.884523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:32.884731 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:32.884879 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:32.884888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.886065 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.886068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:32.886725 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:32.886735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:32.887193 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:32.887196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:32.887300 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:32.887837 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:32.887848 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:32.888176 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:32.890023 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:32.890028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:32.890031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:32.890317 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:32.890606 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:32.890609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:32.890611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:32.890700 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:32.891075 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:32.891078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:32.891179 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 22:26:32.891310 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 22:26:32.891464 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab.
Entry for principal oryx/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab.
I20260619 22:26:32.908636 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33073
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:33.021104 13565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:33.021384 13565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:33.021428 13565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:33.025491 13565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:33.025549 13565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:33.025588 13565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:33.025621 13565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:33.030896 13565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:33.032302 13565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:33.033277 13565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:33.043247 13570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:33.043247 13571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:33.043496 13565 server_base.cc:1034] running on GCE node
W20260619 22:26:33.043285 13573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:33.043931 13565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:33.044683 13565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:33.045871 13565 hybrid_clock.cc:648] HybridClock initialized: now 1781907993045855 us; error 26 us; skew 500 ppm
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907993, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:33.048559 13565 init.cc:377] Logged in from keytab as oryx/127.5.176.194@KRBTEST.COM (short username oryx)
I20260619 22:26:33.049449 13565 webserver.cc:460] Webserver started at http://127.5.176.194:36803/ using document root <none> and password file <none>
I20260619 22:26:33.050007 13565 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:33.050096 13565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:33.050351 13565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:33.052328 13565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
format_stamp: "Formatted at 2026-06-19 22:26:33 on dist-test-slave-bt27"
server_key: "7c5c96eab3a757c469f3f313ac091cda"
server_key_iv: "84c62d70e634f79748e01dde52869095"
server_key_version: "encryptionkey@0"
I20260619 22:26:33.052889 13565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
format_stamp: "Formatted at 2026-06-19 22:26:33 on dist-test-slave-bt27"
server_key: "7c5c96eab3a757c469f3f313ac091cda"
server_key_iv: "84c62d70e634f79748e01dde52869095"
server_key_version: "encryptionkey@0"
I20260619 22:26:33.056624 13565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:26:33.059181 13580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:33.060427 13565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:33.060568 13565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
format_stamp: "Formatted at 2026-06-19 22:26:33 on dist-test-slave-bt27"
server_key: "7c5c96eab3a757c469f3f313ac091cda"
server_key_iv: "84c62d70e634f79748e01dde52869095"
server_key_version: "encryptionkey@0"
I20260619 22:26:33.060688 13565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:33.097992 13565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:33.101334 13565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:33.101565 13565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:33.103245 13565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:33.103325 13565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:33.103391 13565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:33.103427 13565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907993, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:33.115154 13565 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:43473
I20260619 22:26:33.115321 13695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:43473 every 8 connection(s)
I20260619 22:26:33.116333 13565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:33.118368 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.104209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:35625 (local address 127.5.176.254:33073)
0619 22:26:33.104347 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.104352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.105361 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 22:26:33.106479 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.106483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.106547 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.106609 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:33.108260 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.108829 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.110043 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.110360 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.110483 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.113948 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:33.113956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:33.113962 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:33.113994 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:33.115996 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.116626 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.116632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.116636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.116742 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.117150 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.117153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.117155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.117372 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:33.117486 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.118008 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.118186 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 22:26:33.119100 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.104654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.5.176.194:35625)
0619 22:26:33.105184 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.105245 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.106132 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.107097 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.107106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.107477 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:33.108104 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.108119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.108967 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.108970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.109900 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.109910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.110540 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.110544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.110783 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.111528 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:33.111555 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:33.113727 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:33.116144 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.116149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.116160 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.116519 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.116879 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.116882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.116884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.117060 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.117497 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:33.117505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:33.117758 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.117960 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.118306 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":143,"threads_started":1}
I20260619 22:26:33.120419 13696 heartbeater.cc:347] Connected to a master server at 127.5.176.254:33073
I20260619 22:26:33.120585 13696 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:33.121093 13696 heartbeater.cc:511] Master 127.5.176.254:33073 requested a full tablet report, sending...
I20260619 22:26:33.122206 13352 ts_manager.cc:194] Registered new tserver with Master: 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:33.123036 13352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:35625
I20260619 22:26:33.126051 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 13565
I20260619 22:26:33.126220 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:33.126520 5827 external_mini_cluster.cc:1428] Setting key 5676bcc0998d7dee43d9d939862336f0
I20260619 22:26:33.130105 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.122624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:41409 (local address 127.5.176.254:33073)
0619 22:26:33.122745 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.122749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.122883 (+ 134us) server_negotiation.cc:407] Connection header received
0619 22:26:33.122988 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.122990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.123026 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.123091 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:33.123629 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.124373 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.125179 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.125387 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.125525 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.126803 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:33.126807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:33.126810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:33.126836 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:33.128295 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.128814 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.128817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.128818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.128894 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.129189 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.129192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.129193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.129424 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:33.129547 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.129831 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.129941 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:26:33.130123 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.122504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.5.176.194:41409)
0619 22:26:33.122796 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.122811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.122908 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.123136 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.123139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.123373 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:33.123501 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.123509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.124521 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.124523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.125077 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.125084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.125538 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.125541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.125607 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.126216 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:33.126231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:33.126656 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:33.128411 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.128418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.128423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.128700 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.128988 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.128994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.129000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.129102 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.129535 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:33.129539 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:33.129647 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.129759 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.129928 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab.
Entry for principal oryx/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab.
I20260619 22:26:33.156971 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33073
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:33.271564 13705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:33.271829 13705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:33.271874 13705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:33.275955 13705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:33.276014 13705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:33.276053 13705 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 22:26:33.276142 13705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:33.281558 13705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.176.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:33.282708 13705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:33.283752 13705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:33.293521 13713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:33.293561 13705 server_base.cc:1034] running on GCE node
W20260619 22:26:33.293521 13710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:33.293521 13711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:33.294225 13705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:33.294919 13705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:33.296085 13705 hybrid_clock.cc:648] HybridClock initialized: now 1781907993296045 us; error 54 us; skew 500 ppm
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907993, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:33.299043 13705 init.cc:377] Logged in from keytab as oryx/127.5.176.195@KRBTEST.COM (short username oryx)
I20260619 22:26:33.299937 13705 webserver.cc:460] Webserver started at http://127.5.176.195:42283/ using document root <none> and password file <none>
I20260619 22:26:33.300454 13705 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:33.300513 13705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:33.300671 13705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:33.303030 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "ab997f5fb85b47258e30650e4dbe3399"
format_stamp: "Formatted at 2026-06-19 22:26:33 on dist-test-slave-bt27"
server_key: "e2a76cfb5794fea62510a0b2c115ecee"
server_key_iv: "9bfae48b4235853bc509d970d1c537c0"
server_key_version: "encryptionkey@0"
I20260619 22:26:33.303812 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "ab997f5fb85b47258e30650e4dbe3399"
format_stamp: "Formatted at 2026-06-19 22:26:33 on dist-test-slave-bt27"
server_key: "e2a76cfb5794fea62510a0b2c115ecee"
server_key_iv: "9bfae48b4235853bc509d970d1c537c0"
server_key_version: "encryptionkey@0"
I20260619 22:26:33.308071 13705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:33.310588 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:33.311945 13705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:33.312100 13705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "ab997f5fb85b47258e30650e4dbe3399"
format_stamp: "Formatted at 2026-06-19 22:26:33 on dist-test-slave-bt27"
server_key: "e2a76cfb5794fea62510a0b2c115ecee"
server_key_iv: "9bfae48b4235853bc509d970d1c537c0"
server_key_version: "encryptionkey@0"
I20260619 22:26:33.312218 13705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:33.339116 13705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:33.342312 13705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:33.342518 13705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:33.343964 13705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:33.344039 13705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:33.344106 13705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:33.344156 13705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907993, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:33.355124 13705 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:41193
I20260619 22:26:33.355197 13835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:41193 every 8 connection(s)
I20260619 22:26:33.356298 13705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:33.358410 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.345040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:53413 (local address 127.5.176.254:33073)
0619 22:26:33.345198 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.345203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.346018 (+ 815us) server_negotiation.cc:407] Connection header received
0619 22:26:33.347118 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.347133 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.347212 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.347306 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:33.348585 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.349488 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.350813 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.351034 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.351149 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.354410 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:33.354417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:33.354423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:33.354457 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:33.356134 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.356759 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.356764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.356767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.356860 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.357263 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.357267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.357269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.357495 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:33.357578 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.358072 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.358238 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 22:26:33.359172 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.345414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.5.176.195:53413)
0619 22:26:33.345859 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.345908 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.346881 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.347483 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.347490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.347914 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:33.348406 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.348419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.349627 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.349631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.350669 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.350683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.351197 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.351201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.351366 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.352023 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:33.352041 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:33.354209 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:33.356261 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.356266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.356277 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.356608 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.356980 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.356984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.356986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.357145 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.357588 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:33.357592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:33.357811 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.358005 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.358352 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":131,"threads_started":1}
I20260619 22:26:33.360530 13836 heartbeater.cc:347] Connected to a master server at 127.5.176.254:33073
I20260619 22:26:33.360689 13836 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:33.361203 13836 heartbeater.cc:511] Master 127.5.176.254:33073 requested a full tablet report, sending...
I20260619 22:26:33.362340 13351 ts_manager.cc:194] Registered new tserver with Master: ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193)
I20260619 22:26:33.363092 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:53413
I20260619 22:26:33.364501 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 13705
I20260619 22:26:33.364635 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:33.365046 5827 external_mini_cluster.cc:1428] Setting key c88d46d17dbed48c0f3a8a98eb3fc6c4
I20260619 22:26:33.367065 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:26:33.371013 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.363435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:52279 (local address 127.5.176.254:33073)
0619 22:26:33.363693 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.363698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.363713 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:26:33.363750 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.363753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.363820 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.363911 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:33.364416 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.365083 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.366309 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.366606 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.366751 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.367944 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:33.367947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:33.367949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:33.367970 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:33.369322 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.369811 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.369814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.369816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.369902 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.370242 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.370244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.370245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.370413 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:33.370489 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.370795 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.370901 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260619 22:26:33.371186 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.362756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.5.176.195:52279)
0619 22:26:33.363012 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.363025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.363096 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.363921 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.363924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.364131 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:33.364287 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.364302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.365249 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.365252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.366185 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.366195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.366712 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.366715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.366781 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.367358 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:33.367372 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:33.367808 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:33.369428 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.369434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.369438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.369705 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.369994 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:33.370000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:33.370005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:33.370114 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:33.370508 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:33.370514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:33.370624 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.370739 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.370907 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:26:33.391327 13402 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:33.385551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.5.176.254:33073)
0619 22:26:33.385705 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.385710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.385722 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:33.385764 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.385766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.385804 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.385890 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:33.386637 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.387154 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.387951 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.388147 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.388217 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.391213 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44128: BlockingRecv error: recv got EOF from 127.0.0.1:44128 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 22:26:33 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:33.400185 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.392795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.5.176.254:33073)
0619 22:26:33.392926 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.392929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.392942 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:33.393048 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.393050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.393084 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.393147 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:33.393660 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.394209 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.394926 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.395121 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.395192 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.397374 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:33.397376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:33.397379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:33.397400 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:33.398818 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.399173 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.399176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.399177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.399243 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:33.399453 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:33.399455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:33.399457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:33.399616 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:33.399712 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.399885 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.399970 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:26:33.403115 13351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:33.406349 13351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:33.420646 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.415536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.5.176.193:36023)
0619 22:26:33.416033 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.416039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.416058 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:26:33.416118 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.416122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.416263 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.416484 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:33.416849 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.417611 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.419206 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.419985 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.420089 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.420234 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.420457 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.420534 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":83,"threads_started":1}
I20260619 22:26:33.420682 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.414663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:36023 (local address 127.0.0.1:52520)
0619 22:26:33.415199 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.415213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.415312 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.416400 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.416403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.416420 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:33.416708 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.416714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.417720 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.417723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.419106 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.419117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.420140 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.420143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.420259 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.420309 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.420422 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.420538 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":233,"thread_start_us":89,"threads_started":1}
I20260619 22:26:33.421473 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.415222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43473 (local address 127.0.0.1:45942)
0619 22:26:33.415512 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.415525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.415607 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.416359 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.416365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.416399 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:33.416670 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.416676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.417761 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.417763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.418995 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.419004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.421059 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.421063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.421142 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.421162 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.421238 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.421320 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":84,"spinlock_wait_cycles":5248,"thread_start_us":91,"threads_started":1}
I20260619 22:26:33.421991 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.414671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.5.176.195:41193)
0619 22:26:33.415076 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.415090 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.415139 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:26:33.415782 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.415788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.416003 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.416194 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:33.417780 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.419040 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.420168 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.420989 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.421084 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.421236 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.421735 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.421844 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":89,"threads_started":1}
I20260619 22:26:33.421998 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.415391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.5.176.194:43473)
0619 22:26:33.415872 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.415886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.415911 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:26:33.416024 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.416030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.416210 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.416378 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:33.416781 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.417622 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.419115 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.419985 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.420090 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.420248 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.421279 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.421851 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":78,"threads_started":1}
I20260619 22:26:33.421653 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.414553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:41193 (local address 127.0.0.1:43300)
0619 22:26:33.415016 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.415062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.415617 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.416359 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.416372 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.416400 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:33.416655 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.416664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.419169 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.419174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.420015 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.420024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.421287 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.421291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.421375 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.421394 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.421458 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.421527 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":92,"spinlock_wait_cycles":6144,"thread_start_us":74,"threads_started":1}
I20260619 22:26:33.424340 13770 tablet_service.cc:1467] Processing CreateTablet for tablet 27b94853c2cf405d9b5694acaa5722f5 (DEFAULT_TABLE table=test-table [id=b48d2ebc5fb34163ba0cd17c6c5ae70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:33.424156 13490 tablet_service.cc:1467] Processing CreateTablet for tablet 27b94853c2cf405d9b5694acaa5722f5 (DEFAULT_TABLE table=test-table [id=b48d2ebc5fb34163ba0cd17c6c5ae70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:33.425489 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b94853c2cf405d9b5694acaa5722f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:33.425493 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b94853c2cf405d9b5694acaa5722f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:33.426949 13630 tablet_service.cc:1467] Processing CreateTablet for tablet 27b94853c2cf405d9b5694acaa5722f5 (DEFAULT_TABLE table=test-table [id=b48d2ebc5fb34163ba0cd17c6c5ae70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:33.427950 13630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b94853c2cf405d9b5694acaa5722f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:33.431859 13864 tablet_bootstrap.cc:492] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Bootstrap starting.
I20260619 22:26:33.433245 13865 tablet_bootstrap.cc:492] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Bootstrap starting.
I20260619 22:26:33.434474 13864 tablet_bootstrap.cc:654] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:33.435287 13866 tablet_bootstrap.cc:492] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Bootstrap starting.
I20260619 22:26:33.435338 13865 tablet_bootstrap.cc:654] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:33.435338 13864 log.cc:824] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:33.436139 13865 log.cc:824] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:33.437374 13864 tablet_bootstrap.cc:492] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: No bootstrap required, opened a new log
I20260619 22:26:33.437534 13866 tablet_bootstrap.cc:654] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:33.437572 13864 ts_tablet_manager.cc:1397] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:33.437988 13865 tablet_bootstrap.cc:492] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: No bootstrap required, opened a new log
I20260619 22:26:33.438176 13865 ts_tablet_manager.cc:1397] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:26:33.438254 13866 log.cc:824] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:33.439836 13866 tablet_bootstrap.cc:492] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: No bootstrap required, opened a new log
I20260619 22:26:33.439997 13866 ts_tablet_manager.cc:1397] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 22:26:33.442205 13865 raft_consensus.cc:348] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.442205 13864 raft_consensus.cc:348] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.442474 13864 raft_consensus.cc:374] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:33.442473 13865 raft_consensus.cc:374] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:33.442530 13864 raft_consensus.cc:729] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79be34f85714ea19d499cfe703ad1fb, State: Initialized, Role: FOLLOWER
I20260619 22:26:33.442538 13865 raft_consensus.cc:729] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Initialized, Role: FOLLOWER
I20260619 22:26:33.442891 13866 raft_consensus.cc:348] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.442996 13865 consensus_queue.cc:260] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.442996 13864 consensus_queue.cc:260] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.443128 13866 raft_consensus.cc:374] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:33.443183 13866 raft_consensus.cc:729] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6ba8a84944f0592643f8bbd4f5c09, State: Initialized, Role: FOLLOWER
I20260619 22:26:33.443614 13866 consensus_queue.cc:260] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.443949 13864 ts_tablet_manager.cc:1428] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 22:26:33.443949 13865 ts_tablet_manager.cc:1428] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:26:33.444307 13556 heartbeater.cc:503] Master 127.5.176.254:33073 was elected leader, sending a full tablet report...
I20260619 22:26:33.444577 13866 ts_tablet_manager.cc:1428] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260619 22:26:33.444643 13696 heartbeater.cc:503] Master 127.5.176.254:33073 was elected leader, sending a full tablet report...
I20260619 22:26:33.444845 13836 heartbeater.cc:503] Master 127.5.176.254:33073 was elected leader, sending a full tablet report...
W20260619 22:26:33.607475 13837 tablet.cc:2369] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:33.617529 13697 tablet.cc:2369] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:26:33.626058 13557 tablet.cc:2369] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:26:33.628486 13871 raft_consensus.cc:482] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:33.628705 13871 raft_consensus.cc:504] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.629896 13871 leader_election.cc:288] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:33.633982 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.630273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:36023 (local address 127.5.176.195:38707)
0619 22:26:33.630401 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.630418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.630529 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.630861 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.630867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.630888 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:33.631011 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.631018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.631954 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.631958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.632933 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.632942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.633553 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.633557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.633657 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.633682 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.633763 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.633842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 22:26:33.633988 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.630304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:38707 (local address 127.5.176.193:36023)
0619 22:26:33.630472 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.630476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.630493 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:26:33.630677 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.630684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.630748 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.630831 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:33.631167 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.631853 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.633053 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.633462 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.633553 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.633593 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.633781 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.633865 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:26:33.634714 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b94853c2cf405d9b5694acaa5722f5" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb" is_pre_election: true
I20260619 22:26:33.634999 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.630699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:53349 (local address 127.5.176.194:43473)
0619 22:26:33.630837 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.630841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.631137 (+ 296us) server_negotiation.cc:407] Connection header received
0619 22:26:33.631197 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.631200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.631248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.631330 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:33.632079 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.632572 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.633892 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.634485 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.634576 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.634610 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.634855 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.634909 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:33.635066 13510 raft_consensus.cc:2454] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 0.
I20260619 22:26:33.635141 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.630596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43473 (local address 127.5.176.195:53349)
0619 22:26:33.631018 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:33.631031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:33.631124 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:33.631377 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:33.631380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:33.631403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:33.631947 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.631954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.632678 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.632681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.633784 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:33.633791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.634611 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.634615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:33.634715 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.634745 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:26:33.634838 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:26:33.634994 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":138,"threads_started":1}
I20260619 22:26:33.635550 13723 leader_election.cc:302] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab997f5fb85b47258e30650e4dbe3399, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:33.635862 13871 raft_consensus.cc:2781] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:33.635972 13871 raft_consensus.cc:482] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:33.636029 13871 raft_consensus.cc:3037] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:33.636268 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b94853c2cf405d9b5694acaa5722f5" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" is_pre_election: true
I20260619 22:26:33.636538 13650 raft_consensus.cc:2454] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 0.
I20260619 22:26:33.637197 13871 raft_consensus.cc:504] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.637501 13871 leader_election.cc:288] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 election: Requested vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:33.637948 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b94853c2cf405d9b5694acaa5722f5" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
I20260619 22:26:33.637975 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b94853c2cf405d9b5694acaa5722f5" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb"
I20260619 22:26:33.638067 13650 raft_consensus.cc:3037] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:33.638089 13510 raft_consensus.cc:3037] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:33.639393 13650 raft_consensus.cc:2454] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 1.
I20260619 22:26:33.639393 13510 raft_consensus.cc:2454] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 1.
I20260619 22:26:33.639715 13723 leader_election.cc:302] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eb6ba8a84944f0592643f8bbd4f5c09, ab997f5fb85b47258e30650e4dbe3399; no voters:
I20260619 22:26:33.639883 13871 raft_consensus.cc:2781] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:33.640178 13871 raft_consensus.cc:686] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 1 LEADER]: Becoming Leader. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Running, Role: LEADER
I20260619 22:26:33.640592 13871 consensus_queue.cc:237] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } }
I20260619 22:26:33.643847 13352 catalog_manager.cc:5495] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195). New cstate: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:33.655922 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:33.652066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.5.176.195:41193)
0619 22:26:33.652171 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:33.652177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:33.652189 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:33.652366 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:33.652369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:33.652420 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:33.652504 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:33.652714 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.653202 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.654247 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:33.654509 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:33.654615 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:33.655029 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:33.655511 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:33.655608 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:33.655671 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 22:26:33.655730 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 22:26:33.662436 13510 raft_consensus.cc:1261] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Refusing update from remote peer ab997f5fb85b47258e30650e4dbe3399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:33.662452 13650 raft_consensus.cc:1261] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Refusing update from remote peer ab997f5fb85b47258e30650e4dbe3399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:33.663167 13871 consensus_queue.cc:1035] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:33.663405 13874 consensus_queue.cc:1035] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:33.672904 13878 mvcc.cc:205] Tried to move back new op lower bound from 7298695142033281024 to 7298695141955207168. Current Snapshot: MvccSnapshot[applied={T|T < 7298695142033281024}]
I20260619 22:26:33.674590 13879 mvcc.cc:205] Tried to move back new op lower bound from 7298695142033281024 to 7298695141955207168. Current Snapshot: MvccSnapshot[applied={T|T < 7298695142033281024 or (T in {7298695142033281024})}]
I20260619 22:26:33.679095 13351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:33.679286 13351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:33.681655 13351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Sending DeleteTablet for 3 replicas of tablet 27b94853c2cf405d9b5694acaa5722f5
I20260619 22:26:33.682590 13770 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b94853c2cf405d9b5694acaa5722f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:33 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:43300
I20260619 22:26:33.682569 13630 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b94853c2cf405d9b5694acaa5722f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:33 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:45942
I20260619 22:26:33.682628 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b94853c2cf405d9b5694acaa5722f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:33 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52520
I20260619 22:26:33.683126 13889 tablet_replica.cc:330] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: stopping tablet replica
I20260619 22:26:33.683477 13889 raft_consensus.cc:2229] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:33.683843 13889 raft_consensus.cc:2258] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:33.683866 13351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:33.684335 13351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:33.686021 13889 ts_tablet_manager.cc:1905] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:33.688874 13889 ts_tablet_manager.cc:1918] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:33.688975 13889 log.cc:1196] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/27b94853c2cf405d9b5694acaa5722f5
I20260619 22:26:33.689272 13889 ts_tablet_manager.cc:1939] T 27b94853c2cf405d9b5694acaa5722f5 P ab997f5fb85b47258e30650e4dbe3399: Deleting consensus metadata
I20260619 22:26:33.689304 13770 tablet_service.cc:1467] Processing CreateTablet for tablet 31839cb7a1b14ea995c42a3c01b77927 (DEFAULT_TABLE table=test-table [id=de6955e6161a4c318363da0914b09590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:33.689316 13490 tablet_service.cc:1467] Processing CreateTablet for tablet 31839cb7a1b14ea995c42a3c01b77927 (DEFAULT_TABLE table=test-table [id=de6955e6161a4c318363da0914b09590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:33.689563 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31839cb7a1b14ea995c42a3c01b77927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:33.689563 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31839cb7a1b14ea995c42a3c01b77927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:33.690318 13887 tablet_replica.cc:330] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: stopping tablet replica
I20260619 22:26:33.690317 13888 tablet_replica.cc:330] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: stopping tablet replica
I20260619 22:26:33.690637 13887 raft_consensus.cc:2229] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:33.690637 13888 raft_consensus.cc:2229] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:33.690937 13888 raft_consensus.cc:2258] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:33.690954 13887 raft_consensus.cc:2258] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:33.692052 13864 tablet_bootstrap.cc:492] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Bootstrap starting.
I20260619 22:26:33.692126 13337 catalog_manager.cc:4841] TS ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193): tablet 27b94853c2cf405d9b5694acaa5722f5 (table test-table [id=b48d2ebc5fb34163ba0cd17c6c5ae70b]) successfully deleted
I20260619 22:26:33.691778 13865 tablet_bootstrap.cc:492] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Bootstrap starting.
I20260619 22:26:33.692217 13630 tablet_service.cc:1467] Processing CreateTablet for tablet 31839cb7a1b14ea995c42a3c01b77927 (DEFAULT_TABLE table=test-table [id=de6955e6161a4c318363da0914b09590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:33.692480 13630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31839cb7a1b14ea995c42a3c01b77927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:33.693735 13865 tablet_bootstrap.cc:654] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:33.694465 13866 tablet_bootstrap.cc:492] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Bootstrap starting.
I20260619 22:26:33.694990 13864 tablet_bootstrap.cc:654] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:33.695185 13866 tablet_bootstrap.cc:654] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:33.696178 13864 tablet_bootstrap.cc:492] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: No bootstrap required, opened a new log
I20260619 22:26:33.696260 13864 ts_tablet_manager.cc:1397] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 22:26:33.696744 13864 raft_consensus.cc:348] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:33.696887 13864 raft_consensus.cc:374] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:33.696925 13864 raft_consensus.cc:729] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79be34f85714ea19d499cfe703ad1fb, State: Initialized, Role: FOLLOWER
I20260619 22:26:33.696986 13888 ts_tablet_manager.cc:1905] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:33.697043 13864 consensus_queue.cc:260] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:33.697360 13864 ts_tablet_manager.cc:1428] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:33.697939 13865 tablet_bootstrap.cc:492] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: No bootstrap required, opened a new log
I20260619 22:26:33.698014 13866 tablet_bootstrap.cc:492] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: No bootstrap required, opened a new log
I20260619 22:26:33.698024 13865 ts_tablet_manager.cc:1397] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 22:26:33.698091 13866 ts_tablet_manager.cc:1397] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 22:26:33.698354 13887 ts_tablet_manager.cc:1905] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:33.698534 13865 raft_consensus.cc:348] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:33.698632 13865 raft_consensus.cc:374] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:33.698665 13865 raft_consensus.cc:729] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Initialized, Role: FOLLOWER
I20260619 22:26:33.698798 13865 consensus_queue.cc:260] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:33.698915 13866 raft_consensus.cc:348] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:33.699040 13866 raft_consensus.cc:374] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:33.699045 13865 ts_tablet_manager.cc:1428] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 22:26:33.699075 13866 raft_consensus.cc:729] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6ba8a84944f0592643f8bbd4f5c09, State: Initialized, Role: FOLLOWER
I20260619 22:26:33.699189 13866 consensus_queue.cc:260] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:33.699452 13866 ts_tablet_manager.cc:1428] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 22:26:33.702706 13888 ts_tablet_manager.cc:1918] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:33.702805 13888 log.cc:1196] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/27b94853c2cf405d9b5694acaa5722f5
I20260619 22:26:33.703161 13888 ts_tablet_manager.cc:1939] T 27b94853c2cf405d9b5694acaa5722f5 P e79be34f85714ea19d499cfe703ad1fb: Deleting consensus metadata
I20260619 22:26:33.703207 13887 ts_tablet_manager.cc:1918] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:33.703289 13887 log.cc:1196] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/27b94853c2cf405d9b5694acaa5722f5
I20260619 22:26:33.703576 13887 ts_tablet_manager.cc:1939] T 27b94853c2cf405d9b5694acaa5722f5 P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting consensus metadata
I20260619 22:26:33.704455 13339 catalog_manager.cc:4841] TS e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023): tablet 27b94853c2cf405d9b5694acaa5722f5 (table test-table [id=b48d2ebc5fb34163ba0cd17c6c5ae70b]) successfully deleted
I20260619 22:26:33.704633 13339 catalog_manager.cc:4841] TS 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473): tablet 27b94853c2cf405d9b5694acaa5722f5 (table test-table [id=b48d2ebc5fb34163ba0cd17c6c5ae70b]) successfully deleted
I20260619 22:26:34.036413 13872 raft_consensus.cc:482] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:34.036640 13872 raft_consensus.cc:504] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.037858 13872 leader_election.cc:288] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193)
I20260619 22:26:34.042074 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.038321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:44303 (local address 127.5.176.195:41193)
0619 22:26:34.038468 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.038474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.038853 (+ 379us) server_negotiation.cc:407] Connection header received
0619 22:26:34.038978 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.038982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.039031 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.039112 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:34.039444 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.039949 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.040960 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.041383 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.041472 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.041512 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.041885 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.041946 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:26:34.042075 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.038393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52851 (local address 127.5.176.193:36023)
0619 22:26:34.038528 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.038534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.038669 (+ 135us) server_negotiation.cc:407] Connection header received
0619 22:26:34.038780 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.038786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.038851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.038941 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:34.039255 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.039848 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.041053 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.041489 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.041617 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.041669 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.041860 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.041935 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:26:34.042093 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.038173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:36023 (local address 127.5.176.194:52851)
0619 22:26:34.038553 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.038567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.038684 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.038971 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.038973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.038991 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:34.039136 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.039143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.039965 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.039971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.040928 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.040939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.041615 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.041617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.041717 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.041753 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.041810 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.041910 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":19,"thread_start_us":102,"threads_started":1}
I20260619 22:26:34.042104 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.038173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:41193 (local address 127.5.176.194:44303)
0619 22:26:34.038730 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.038746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.038867 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.039139 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.039141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.039165 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:34.039325 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.039333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.040068 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.040070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.040842 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.040851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.041522 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.041529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.041638 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.041749 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.041859 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.041935 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":143,"thread_start_us":104,"threads_started":1}
I20260619 22:26:34.042811 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31839cb7a1b14ea995c42a3c01b77927" candidate_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb" is_pre_election: true
I20260619 22:26:34.042860 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31839cb7a1b14ea995c42a3c01b77927" candidate_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab997f5fb85b47258e30650e4dbe3399" is_pre_election: true
I20260619 22:26:34.042984 13510 raft_consensus.cc:2454] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eb6ba8a84944f0592643f8bbd4f5c09 in term 0.
I20260619 22:26:34.043118 13790 raft_consensus.cc:2454] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eb6ba8a84944f0592643f8bbd4f5c09 in term 0.
I20260619 22:26:34.043371 13583 leader_election.cc:302] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eb6ba8a84944f0592643f8bbd4f5c09, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:34.043625 13872 raft_consensus.cc:2781] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:34.043696 13872 raft_consensus.cc:482] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:34.043747 13872 raft_consensus.cc:3037] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.044785 13872 raft_consensus.cc:504] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.045209 13872 leader_election.cc:288] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [CANDIDATE]: Term 1 election: Requested vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193)
I20260619 22:26:34.045547 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31839cb7a1b14ea995c42a3c01b77927" candidate_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb"
I20260619 22:26:34.045606 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31839cb7a1b14ea995c42a3c01b77927" candidate_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab997f5fb85b47258e30650e4dbe3399"
I20260619 22:26:34.045691 13510 raft_consensus.cc:3037] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.045740 13790 raft_consensus.cc:3037] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.046708 13510 raft_consensus.cc:2454] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eb6ba8a84944f0592643f8bbd4f5c09 in term 1.
I20260619 22:26:34.046850 13790 raft_consensus.cc:2454] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eb6ba8a84944f0592643f8bbd4f5c09 in term 1.
I20260619 22:26:34.047029 13583 leader_election.cc:302] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eb6ba8a84944f0592643f8bbd4f5c09, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:34.047231 13872 raft_consensus.cc:2781] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:34.047457 13872 raft_consensus.cc:686] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 LEADER]: Becoming Leader. State: Replica: 5eb6ba8a84944f0592643f8bbd4f5c09, State: Running, Role: LEADER
I20260619 22:26:34.047704 13872 consensus_queue.cc:237] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.049780 13351 catalog_manager.cc:5495] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } health_report { overall_health: UNKNOWN } } }
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907992, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.254@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:34.088858 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.080212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.5.176.254:33073)
0619 22:26:34.080531 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.080541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.080672 (+ 131us) server_negotiation.cc:407] Connection header received
0619 22:26:34.080839 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.080843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.080891 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.080990 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:34.081674 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.082297 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.083083 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.083320 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.083430 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.085773 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:34.085776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:34.085778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:34.085801 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:34.087130 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:34.087584 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:34.087589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:34.087591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:34.087672 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:34.087950 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:34.087953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:34.087955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:34.088134 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:34.088239 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.088520 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.088731 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":269,"thread_start_us":99,"threads_started":1}
I20260619 22:26:34.088857 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.080087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33073 (local address 127.0.0.1:44138)
0619 22:26:34.080548 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.080564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.080737 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.081005 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.081008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.081441 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:34.081547 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.081554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.082403 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.082409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.082969 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.082976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.083433 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.083437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.083504 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.084124 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:34.084137 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:34.085627 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:34.087240 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:34.087251 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:34.087258 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:34.087488 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:34.087768 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:34.087771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:34.087773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:34.087858 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:34.088239 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:34.088247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:34.088361 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.088477 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.088643 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260619 22:26:34.092813 13351 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:44138:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 22:26:34.093549 13351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:34.099153 13770 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbfd47c79434a65af98993d2d84df4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a94d20646d0f48e88c6c8fb4bd1897ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:34.099165 13490 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbfd47c79434a65af98993d2d84df4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a94d20646d0f48e88c6c8fb4bd1897ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:34.099181 13630 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbfd47c79434a65af98993d2d84df4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a94d20646d0f48e88c6c8fb4bd1897ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 22:26:34.099596 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbfd47c79434a65af98993d2d84df4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.099606 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbfd47c79434a65af98993d2d84df4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.099627 13630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbfd47c79434a65af98993d2d84df4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.102732 13865 tablet_bootstrap.cc:492] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: Bootstrap starting.
I20260619 22:26:34.103446 13864 tablet_bootstrap.cc:492] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: Bootstrap starting.
I20260619 22:26:34.103967 13865 tablet_bootstrap.cc:654] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.104458 13864 tablet_bootstrap.cc:654] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.105556 13866 tablet_bootstrap.cc:492] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: Bootstrap starting.
I20260619 22:26:34.105758 13864 tablet_bootstrap.cc:492] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: No bootstrap required, opened a new log
I20260619 22:26:34.105834 13864 ts_tablet_manager.cc:1397] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 22:26:34.106312 13865 tablet_bootstrap.cc:492] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: No bootstrap required, opened a new log
I20260619 22:26:34.106393 13865 ts_tablet_manager.cc:1397] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 22:26:34.106331 13864 raft_consensus.cc:348] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.106447 13864 raft_consensus.cc:374] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.106489 13864 raft_consensus.cc:729] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79be34f85714ea19d499cfe703ad1fb, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.106571 13866 tablet_bootstrap.cc:654] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.106632 13864 consensus_queue.cc:260] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.106961 13864 ts_tablet_manager.cc:1428] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:34.106889 13865 raft_consensus.cc:348] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.107017 13865 raft_consensus.cc:374] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.107059 13865 raft_consensus.cc:729] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.107244 13865 consensus_queue.cc:260] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.107520 13874 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.107677 13874 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.108371 13865 ts_tablet_manager.cc:1428] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 22:26:34.108624 13870 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.108801 13870 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.110265 13866 tablet_bootstrap.cc:492] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: No bootstrap required, opened a new log
I20260619 22:26:34.110365 13866 ts_tablet_manager.cc:1397] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 22:26:34.110800 13866 raft_consensus.cc:348] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.110895 13866 raft_consensus.cc:374] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.110917 13866 raft_consensus.cc:729] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6ba8a84944f0592643f8bbd4f5c09, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.111068 13866 consensus_queue.cc:260] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.111317 13866 ts_tablet_manager.cc:1428] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 22:26:34.111318 13872 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.111471 13872 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.173666 13874 raft_consensus.cc:482] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:34.173818 13874 raft_consensus.cc:504] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.174276 13874 leader_election.cc:288] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:34.174747 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfd47c79434a65af98993d2d84df4d" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb" is_pre_election: true
I20260619 22:26:34.174757 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfd47c79434a65af98993d2d84df4d" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" is_pre_election: true
I20260619 22:26:34.174912 13650 raft_consensus.cc:2454] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 0.
I20260619 22:26:34.174912 13510 raft_consensus.cc:2454] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 0.
I20260619 22:26:34.175194 13723 leader_election.cc:302] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab997f5fb85b47258e30650e4dbe3399, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:34.175498 13874 raft_consensus.cc:2781] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:34.175566 13874 raft_consensus.cc:482] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:34.175621 13874 raft_consensus.cc:3037] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.177098 13874 raft_consensus.cc:504] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.177510 13874 leader_election.cc:288] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 election: Requested vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:34.177948 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfd47c79434a65af98993d2d84df4d" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
I20260619 22:26:34.178085 13650 raft_consensus.cc:3037] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.178272 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfd47c79434a65af98993d2d84df4d" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb"
I20260619 22:26:34.178416 13510 raft_consensus.cc:3037] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.179093 13650 raft_consensus.cc:2454] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 1.
I20260619 22:26:34.179399 13723 leader_election.cc:302] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eb6ba8a84944f0592643f8bbd4f5c09, ab997f5fb85b47258e30650e4dbe3399; no voters:
I20260619 22:26:34.179461 13510 raft_consensus.cc:2454] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 1.
I20260619 22:26:34.179639 13874 raft_consensus.cc:2781] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:34.179867 13874 raft_consensus.cc:686] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [term 1 LEADER]: Becoming Leader. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Running, Role: LEADER
I20260619 22:26:34.180030 13874 consensus_queue.cc:237] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.180658 13907 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: TxnStatusTablet state changed. Reason: New leader ab997f5fb85b47258e30650e4dbe3399. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.180783 13907 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:34.181155 13909 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 22:26:34.181315 13909 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 22:26:34.181844 13352 catalog_manager.cc:5495] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195). New cstate: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } health_report { overall_health: HEALTHY } } }
I20260619 22:26:34.198268 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.194290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.5.176.195:41193)
0619 22:26:34.194429 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.194433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.194516 (+ 83us) server_negotiation.cc:407] Connection header received
0619 22:26:34.194692 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.194695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.194749 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.194828 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.195122 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.195622 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.196838 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.197052 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.197136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.197488 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.197566 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.197732 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.198027 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.198080 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:26:34.198292 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.194167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:41193 (local address 127.0.0.1:43318)
0619 22:26:34.194431 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.194457 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.194576 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.194873 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.194876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.194885 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:34.195018 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.195024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.195723 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.195727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.196687 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.196698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.197164 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.197167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.197265 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.197330 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.197910 (+ 580us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.197928 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.198004 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.198099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 22:26:34.201090 13650 raft_consensus.cc:1261] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Refusing update from remote peer ab997f5fb85b47258e30650e4dbe3399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.201181 13510 raft_consensus.cc:1261] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Refusing update from remote peer ab997f5fb85b47258e30650e4dbe3399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.201503 13907 consensus_queue.cc:1035] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.201668 13874 consensus_queue.cc:1035] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.203118 13872 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: TxnStatusTablet state changed. Reason: New leader ab997f5fb85b47258e30650e4dbe3399. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.203243 13872 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.203416 13870 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: TxnStatusTablet state changed. Reason: New leader ab997f5fb85b47258e30650e4dbe3399. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.203533 13870 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.206302 13907 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.206426 13907 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:34.207118 13872 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.207222 13872 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P 5eb6ba8a84944f0592643f8bbd4f5c09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.208024 13874 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.208132 13874 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P ab997f5fb85b47258e30650e4dbe3399: This TxnStatusTablet replica's current role is: LEADER
I20260619 22:26:34.209002 13870 tablet_replica.cc:439] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } } }
I20260619 22:26:34.209113 13870 tablet_replica.cc:442] T 3bbfd47c79434a65af98993d2d84df4d P e79be34f85714ea19d499cfe703ad1fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 22:26:34.218385 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.214062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.5.176.194:43473)
0619 22:26:34.214331 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.214336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.214349 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:34.214523 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.214526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.214589 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.214710 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.214978 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.215583 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.216505 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.216819 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.216928 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.217323 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.217823 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.217987 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.218081 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.218188 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260619 22:26:34.229943 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.226203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:36213 (local address 127.5.176.195:41193)
0619 22:26:34.226350 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.226355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.226613 (+ 258us) server_negotiation.cc:407] Connection header received
0619 22:26:34.226752 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.226755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.226805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.226876 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.227174 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.227692 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.228850 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.229054 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.229130 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.229486 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.229570 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.229645 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.229763 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.229820 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:26:34.229993 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.226093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:41193 (local address 127.5.176.194:36213)
0619 22:26:34.226496 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.226512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.226630 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.226910 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.226913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.226920 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:34.227035 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.227041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.227803 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.227806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.228733 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.228745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.229163 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.229167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.229255 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.229321 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.229668 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.229675 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.229742 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.229835 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":132,"threads_started":1}
I20260619 22:26:34.238580 13510 raft_consensus.cc:1261] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Refusing update from remote peer 5eb6ba8a84944f0592643f8bbd4f5c09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.238824 13790 raft_consensus.cc:1261] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Refusing update from remote peer 5eb6ba8a84944f0592643f8bbd4f5c09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.239110 13872 consensus_queue.cc:1035] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.239409 13895 consensus_queue.cc:1035] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.244779 13912 mvcc.cc:205] Tried to move back new op lower bound from 7298695144396267520 to 7298695143621554176. Current Snapshot: MvccSnapshot[applied={T|T < 7298695144396267520}]
I20260619 22:26:34.265512 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.260827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43473 (local address 127.5.176.195:44527)
0619 22:26:34.261260 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.261278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.261403 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.261735 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.261737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.261745 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:34.261843 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.261848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.262929 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.262933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.263975 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.263985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.264637 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.264641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.264727 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.264791 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.265198 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.265208 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.265280 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.265368 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":109,"threads_started":1}
I20260619 22:26:34.265506 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.260885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:44527 (local address 127.5.176.194:43473)
0619 22:26:34.261019 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.261025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.261359 (+ 334us) server_negotiation.cc:407] Connection header received
0619 22:26:34.261528 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.261531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.261626 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.261702 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.261946 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.262801 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.264121 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.264503 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.264641 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.264961 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.265121 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.265192 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.265289 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.265346 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:26:34.291610 13352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:34.291796 13352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:34.294266 13352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Sending DeleteTablet for 3 replicas of tablet 31839cb7a1b14ea995c42a3c01b77927
I20260619 22:26:34.294975 13770 tablet_service.cc:1514] Processing DeleteTablet for tablet 31839cb7a1b14ea995c42a3c01b77927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:43300
I20260619 22:26:34.295007 13630 tablet_service.cc:1514] Processing DeleteTablet for tablet 31839cb7a1b14ea995c42a3c01b77927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:45942
I20260619 22:26:34.295361 13933 tablet_replica.cc:330] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: stopping tablet replica
I20260619 22:26:34.295488 13933 raft_consensus.cc:2229] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:34.295598 13933 raft_consensus.cc:2258] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.295611 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet 31839cb7a1b14ea995c42a3c01b77927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52520
I20260619 22:26:34.296386 13934 tablet_replica.cc:330] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: stopping tablet replica
I20260619 22:26:34.296602 13935 tablet_replica.cc:330] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: stopping tablet replica
I20260619 22:26:34.299109 13935 raft_consensus.cc:2229] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:34.299299 13934 raft_consensus.cc:2229] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:34.299670 13934 raft_consensus.cc:2258] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.299867 13935 raft_consensus.cc:2258] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.299924 13933 ts_tablet_manager.cc:1905] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.300770 13935 ts_tablet_manager.cc:1905] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.301268 13934 ts_tablet_manager.cc:1905] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.303280 13933 ts_tablet_manager.cc:1918] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:34.303416 13933 log.cc:1196] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/31839cb7a1b14ea995c42a3c01b77927
I20260619 22:26:34.303766 13933 ts_tablet_manager.cc:1939] T 31839cb7a1b14ea995c42a3c01b77927 P ab997f5fb85b47258e30650e4dbe3399: Deleting consensus metadata
I20260619 22:26:34.304641 13337 catalog_manager.cc:4841] TS ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193): tablet 31839cb7a1b14ea995c42a3c01b77927 (table test-table [id=de6955e6161a4c318363da0914b09590]) successfully deleted
I20260619 22:26:34.306336 13934 ts_tablet_manager.cc:1918] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:34.307366 13935 ts_tablet_manager.cc:1918] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:34.307516 13934 log.cc:1196] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/31839cb7a1b14ea995c42a3c01b77927
I20260619 22:26:34.308100 13935 log.cc:1196] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/31839cb7a1b14ea995c42a3c01b77927
I20260619 22:26:34.308455 13935 ts_tablet_manager.cc:1939] T 31839cb7a1b14ea995c42a3c01b77927 P e79be34f85714ea19d499cfe703ad1fb: Deleting consensus metadata
I20260619 22:26:34.307853 13934 ts_tablet_manager.cc:1939] T 31839cb7a1b14ea995c42a3c01b77927 P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting consensus metadata
I20260619 22:26:34.309383 13339 catalog_manager.cc:4841] TS 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473): tablet 31839cb7a1b14ea995c42a3c01b77927 (table test-table [id=de6955e6161a4c318363da0914b09590]) successfully deleted
I20260619 22:26:34.310235 13339 catalog_manager.cc:4841] TS e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023): tablet 31839cb7a1b14ea995c42a3c01b77927 (table test-table [id=de6955e6161a4c318363da0914b09590]) successfully deleted
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:26:34.340005 13903 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:34.331898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.5.176.254:33073)
0619 22:26:34.332035 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.332040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.332327 (+ 287us) server_negotiation.cc:407] Connection header received
0619 22:26:34.332458 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.332462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.332520 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.332612 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:34.333490 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.334379 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.335588 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.335960 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.336068 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.339865 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44146: BlockingRecv error: recv got EOF from 127.0.0.1:44146 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM
I20260619 22:26:34.353051 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.341691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.5.176.254:33073)
0619 22:26:34.341809 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.341813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.342281 (+ 468us) server_negotiation.cc:407] Connection header received
0619 22:26:34.342416 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.342419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.342472 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.342569 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:34.343480 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.344564 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.345738 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.346094 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.346251 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.349357 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:34.349361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:34.349365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:34.349397 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:34.351155 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:34.351709 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:34.351712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:34.351714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:34.351801 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:34.352104 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:34.352107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:34.352109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:34.352348 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:34.352479 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.352741 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.352874 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:26:34.356370 13352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:34.357070 13352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:34.362565 13770 tablet_service.cc:1467] Processing CreateTablet for tablet e0947db368eb42dd8fbfe39b19f610bd (DEFAULT_TABLE table=test-table [id=27a094c9067f457b91242224d6dab66d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:34.362905 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0947db368eb42dd8fbfe39b19f610bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.362852 13490 tablet_service.cc:1467] Processing CreateTablet for tablet e0947db368eb42dd8fbfe39b19f610bd (DEFAULT_TABLE table=test-table [id=27a094c9067f457b91242224d6dab66d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:34.363130 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0947db368eb42dd8fbfe39b19f610bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.365679 13630 tablet_service.cc:1467] Processing CreateTablet for tablet e0947db368eb42dd8fbfe39b19f610bd (DEFAULT_TABLE table=test-table [id=27a094c9067f457b91242224d6dab66d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:34.365958 13630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0947db368eb42dd8fbfe39b19f610bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.366317 13864 tablet_bootstrap.cc:492] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Bootstrap starting.
I20260619 22:26:34.367326 13864 tablet_bootstrap.cc:654] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.367928 13865 tablet_bootstrap.cc:492] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Bootstrap starting.
I20260619 22:26:34.368323 13866 tablet_bootstrap.cc:492] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Bootstrap starting.
I20260619 22:26:34.368709 13865 tablet_bootstrap.cc:654] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.369246 13866 tablet_bootstrap.cc:654] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.370633 13865 tablet_bootstrap.cc:492] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: No bootstrap required, opened a new log
I20260619 22:26:34.370667 13866 tablet_bootstrap.cc:492] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: No bootstrap required, opened a new log
I20260619 22:26:34.370719 13865 ts_tablet_manager.cc:1397] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 22:26:34.370744 13866 ts_tablet_manager.cc:1397] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:34.371114 13865 raft_consensus.cc:348] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.371166 13866 raft_consensus.cc:348] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.371248 13865 raft_consensus.cc:374] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.371268 13866 raft_consensus.cc:374] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.371280 13865 raft_consensus.cc:729] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.371313 13866 raft_consensus.cc:729] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6ba8a84944f0592643f8bbd4f5c09, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.371429 13866 consensus_queue.cc:260] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.371421 13865 consensus_queue.cc:260] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.371672 13866 ts_tablet_manager.cc:1428] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:34.371685 13865 ts_tablet_manager.cc:1428] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 22:26:34.372351 13864 tablet_bootstrap.cc:492] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: No bootstrap required, opened a new log
I20260619 22:26:34.372460 13864 ts_tablet_manager.cc:1397] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 22:26:34.372936 13864 raft_consensus.cc:348] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.373087 13864 raft_consensus.cc:374] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.373121 13864 raft_consensus.cc:729] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79be34f85714ea19d499cfe703ad1fb, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.373257 13864 consensus_queue.cc:260] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.373548 13864 ts_tablet_manager.cc:1428] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:34.400733 13874 raft_consensus.cc:482] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:34.400882 13874 raft_consensus.cc:504] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.401319 13874 leader_election.cc:288] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:34.401861 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0947db368eb42dd8fbfe39b19f610bd" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" is_pre_election: true
I20260619 22:26:34.402027 13650 raft_consensus.cc:2454] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 0.
I20260619 22:26:34.402392 13723 leader_election.cc:302] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eb6ba8a84944f0592643f8bbd4f5c09, ab997f5fb85b47258e30650e4dbe3399; no voters:
I20260619 22:26:34.402423 13509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0947db368eb42dd8fbfe39b19f610bd" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb" is_pre_election: true
I20260619 22:26:34.402580 13874 raft_consensus.cc:2781] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:34.402590 13509 raft_consensus.cc:2454] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 0.
I20260619 22:26:34.402632 13874 raft_consensus.cc:482] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:34.402696 13874 raft_consensus.cc:3037] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.404125 13874 raft_consensus.cc:504] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.404536 13874 leader_election.cc:288] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 election: Requested vote from peers e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023), 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473)
I20260619 22:26:34.404896 13509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0947db368eb42dd8fbfe39b19f610bd" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79be34f85714ea19d499cfe703ad1fb"
I20260619 22:26:34.404897 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0947db368eb42dd8fbfe39b19f610bd" candidate_uuid: "ab997f5fb85b47258e30650e4dbe3399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
I20260619 22:26:34.405021 13650 raft_consensus.cc:3037] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.405042 13509 raft_consensus.cc:3037] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.406198 13509 raft_consensus.cc:2454] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 1.
I20260619 22:26:34.406198 13650 raft_consensus.cc:2454] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab997f5fb85b47258e30650e4dbe3399 in term 1.
I20260619 22:26:34.406476 13723 leader_election.cc:302] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab997f5fb85b47258e30650e4dbe3399, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:34.406648 13874 raft_consensus.cc:2781] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:34.406764 13874 raft_consensus.cc:686] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 1 LEADER]: Becoming Leader. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Running, Role: LEADER
I20260619 22:26:34.406919 13874 consensus_queue.cc:237] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.408665 13352 catalog_manager.cc:5495] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195). New cstate: current_term: 1 leader_uuid: "ab997f5fb85b47258e30650e4dbe3399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } health_report { overall_health: HEALTHY } } }
I20260619 22:26:34.419922 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.416024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.5.176.195:41193)
0619 22:26:34.416363 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.416368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.416380 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:34.416425 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.416428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.416481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.416557 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.416918 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.417702 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.418621 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.418977 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.419089 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.419304 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.419396 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.419471 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.419634 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.419701 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260619 22:26:34.422041 13509 raft_consensus.cc:1261] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Refusing update from remote peer ab997f5fb85b47258e30650e4dbe3399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.422041 13650 raft_consensus.cc:1261] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Refusing update from remote peer ab997f5fb85b47258e30650e4dbe3399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.422564 13874 consensus_queue.cc:1035] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.422720 13907 consensus_queue.cc:1035] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.431732 13348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:34.431891 13348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:34.433913 13348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Sending DeleteTablet for 3 replicas of tablet e0947db368eb42dd8fbfe39b19f610bd
I20260619 22:26:34.434531 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet e0947db368eb42dd8fbfe39b19f610bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52520
I20260619 22:26:34.434547 13770 tablet_service.cc:1514] Processing DeleteTablet for tablet e0947db368eb42dd8fbfe39b19f610bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:43300
I20260619 22:26:34.434703 13933 tablet_replica.cc:330] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: stopping tablet replica
I20260619 22:26:34.434702 13935 tablet_replica.cc:330] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: stopping tablet replica
I20260619 22:26:34.434808 13935 raft_consensus.cc:2229] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:34.434820 13933 raft_consensus.cc:2229] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:34.434945 13935 raft_consensus.cc:2258] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.435001 13933 raft_consensus.cc:2258] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.435145 13348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 22:26:34.435343 13630 tablet_service.cc:1514] Processing DeleteTablet for tablet e0947db368eb42dd8fbfe39b19f610bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:45942
I20260619 22:26:34.435514 13934 tablet_replica.cc:330] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: stopping tablet replica
W20260619 22:26:34.435575 13348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:34.435624 13934 raft_consensus.cc:2229] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:34.435738 13934 raft_consensus.cc:2258] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.436025 13933 ts_tablet_manager.cc:1905] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.436394 13934 ts_tablet_manager.cc:1905] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.438062 13933 ts_tablet_manager.cc:1918] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:34.438128 13933 log.cc:1196] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/e0947db368eb42dd8fbfe39b19f610bd
I20260619 22:26:34.438453 13933 ts_tablet_manager.cc:1939] T e0947db368eb42dd8fbfe39b19f610bd P ab997f5fb85b47258e30650e4dbe3399: Deleting consensus metadata
I20260619 22:26:34.438966 13934 ts_tablet_manager.cc:1918] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:34.439033 13934 log.cc:1196] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/e0947db368eb42dd8fbfe39b19f610bd
I20260619 22:26:34.439183 13337 catalog_manager.cc:4841] TS ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193): tablet e0947db368eb42dd8fbfe39b19f610bd (table test-table [id=27a094c9067f457b91242224d6dab66d]) successfully deleted
I20260619 22:26:34.440021 13935 ts_tablet_manager.cc:1905] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.442541 13935 ts_tablet_manager.cc:1918] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:34.442634 13935 log.cc:1196] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/e0947db368eb42dd8fbfe39b19f610bd
I20260619 22:26:34.442914 13935 ts_tablet_manager.cc:1939] T e0947db368eb42dd8fbfe39b19f610bd P e79be34f85714ea19d499cfe703ad1fb: Deleting consensus metadata
I20260619 22:26:34.443594 13339 catalog_manager.cc:4841] TS e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023): tablet e0947db368eb42dd8fbfe39b19f610bd (table test-table [id=27a094c9067f457b91242224d6dab66d]) successfully deleted
I20260619 22:26:34.444023 13934 ts_tablet_manager.cc:1939] T e0947db368eb42dd8fbfe39b19f610bd P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting consensus metadata
I20260619 22:26:34.444830 13339 catalog_manager.cc:4841] TS 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473): tablet e0947db368eb42dd8fbfe39b19f610bd (table test-table [id=27a094c9067f457b91242224d6dab66d]) successfully deleted
I20260619 22:26:34.445183 13490 tablet_service.cc:1467] Processing CreateTablet for tablet 1c548efe8f7643deb9a99da16d017b9f (DEFAULT_TABLE table=test-table [id=606c04e2f1894f6bb986a276474878c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:34.445237 13770 tablet_service.cc:1467] Processing CreateTablet for tablet 1c548efe8f7643deb9a99da16d017b9f (DEFAULT_TABLE table=test-table [id=606c04e2f1894f6bb986a276474878c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:34.445447 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c548efe8f7643deb9a99da16d017b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.445544 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c548efe8f7643deb9a99da16d017b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.447394 13865 tablet_bootstrap.cc:492] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Bootstrap starting.
I20260619 22:26:34.447768 13864 tablet_bootstrap.cc:492] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Bootstrap starting.
I20260619 22:26:34.448189 13865 tablet_bootstrap.cc:654] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.448990 13630 tablet_service.cc:1467] Processing CreateTablet for tablet 1c548efe8f7643deb9a99da16d017b9f (DEFAULT_TABLE table=test-table [id=606c04e2f1894f6bb986a276474878c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:34.449214 13865 tablet_bootstrap.cc:492] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: No bootstrap required, opened a new log
I20260619 22:26:34.449252 13630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c548efe8f7643deb9a99da16d017b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:34.449327 13865 ts_tablet_manager.cc:1397] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:34.449669 13865 raft_consensus.cc:348] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.449931 13865 raft_consensus.cc:374] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.449976 13865 raft_consensus.cc:729] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab997f5fb85b47258e30650e4dbe3399, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.450115 13865 consensus_queue.cc:260] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.450407 13865 ts_tablet_manager.cc:1428] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:34.451082 13864 tablet_bootstrap.cc:654] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.454985 13864 tablet_bootstrap.cc:492] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: No bootstrap required, opened a new log
I20260619 22:26:34.454995 13866 tablet_bootstrap.cc:492] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Bootstrap starting.
I20260619 22:26:34.455092 13864 ts_tablet_manager.cc:1397] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 22:26:34.455641 13864 raft_consensus.cc:348] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.455776 13864 raft_consensus.cc:374] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.455813 13864 raft_consensus.cc:729] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79be34f85714ea19d499cfe703ad1fb, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.455983 13866 tablet_bootstrap.cc:654] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:34.455955 13864 consensus_queue.cc:260] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.456218 13864 ts_tablet_manager.cc:1428] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:34.457212 13866 tablet_bootstrap.cc:492] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: No bootstrap required, opened a new log
I20260619 22:26:34.457304 13866 ts_tablet_manager.cc:1397] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 22:26:34.457724 13866 raft_consensus.cc:348] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.457840 13866 raft_consensus.cc:374] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:34.457890 13866 raft_consensus.cc:729] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6ba8a84944f0592643f8bbd4f5c09, State: Initialized, Role: FOLLOWER
I20260619 22:26:34.458010 13866 consensus_queue.cc:260] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.458627 13866 ts_tablet_manager.cc:1428] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:26:34.550194 13870 raft_consensus.cc:482] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:34.550362 13870 raft_consensus.cc:504] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.551463 13870 leader_election.cc:288] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473), ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193)
I20260619 22:26:34.556793 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.551869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:33737 (local address 127.5.176.195:41193)
0619 22:26:34.552037 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.552043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.552146 (+ 103us) server_negotiation.cc:407] Connection header received
0619 22:26:34.552306 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.552309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.552358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.552442 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:34.552832 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.553593 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.555461 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.556100 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.556228 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.556281 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.556570 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.556655 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:26:34.556793 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.551855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:50257 (local address 127.5.176.194:43473)
0619 22:26:34.552039 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.552044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.552215 (+ 171us) server_negotiation.cc:407] Connection header received
0619 22:26:34.552310 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.552313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.552358 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.552442 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:34.552803 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.553594 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.555495 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.556109 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.556229 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.556295 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.556582 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.556666 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 22:26:34.557288 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.551726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43473 (local address 127.5.176.193:50257)
0619 22:26:34.552138 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.552150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.552228 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.552467 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.552470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.552495 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:34.552687 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.552694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.553710 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.553714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.555311 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.555323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.556228 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.556232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.556385 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.556465 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.556950 (+ 485us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.557091 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":326,"spinlock_wait_cycles":384,"thread_start_us":80,"threads_started":1}
I20260619 22:26:34.557297 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.551726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:41193 (local address 127.5.176.193:33737)
0619 22:26:34.552045 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.552061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.552184 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.552467 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.552470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.552500 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:34.552698 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.552703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.553716 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.553719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.555317 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.555323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.556228 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.556232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.556395 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.556431 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.556950 (+ 519us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.557091 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":226,"spinlock_wait_cycles":8448,"thread_start_us":73,"threads_started":1}
I20260619 22:26:34.557871 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c548efe8f7643deb9a99da16d017b9f" candidate_uuid: "e79be34f85714ea19d499cfe703ad1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" is_pre_election: true
I20260619 22:26:34.557925 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c548efe8f7643deb9a99da16d017b9f" candidate_uuid: "e79be34f85714ea19d499cfe703ad1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab997f5fb85b47258e30650e4dbe3399" is_pre_election: true
I20260619 22:26:34.558053 13650 raft_consensus.cc:2454] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79be34f85714ea19d499cfe703ad1fb in term 0.
I20260619 22:26:34.558068 13790 raft_consensus.cc:2454] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79be34f85714ea19d499cfe703ad1fb in term 0.
I20260619 22:26:34.558486 13443 leader_election.cc:302] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eb6ba8a84944f0592643f8bbd4f5c09, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:34.558799 13870 raft_consensus.cc:2781] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:34.558871 13870 raft_consensus.cc:482] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:34.558919 13870 raft_consensus.cc:3037] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.559937 13870 raft_consensus.cc:504] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.560350 13870 leader_election.cc:288] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [CANDIDATE]: Term 1 election: Requested vote from peers 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473), ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193)
I20260619 22:26:34.560761 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c548efe8f7643deb9a99da16d017b9f" candidate_uuid: "e79be34f85714ea19d499cfe703ad1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab997f5fb85b47258e30650e4dbe3399"
I20260619 22:26:34.560876 13790 raft_consensus.cc:3037] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.561692 13790 raft_consensus.cc:2454] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79be34f85714ea19d499cfe703ad1fb in term 1.
I20260619 22:26:34.561934 13650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c548efe8f7643deb9a99da16d017b9f" candidate_uuid: "e79be34f85714ea19d499cfe703ad1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09"
I20260619 22:26:34.562034 13441 leader_election.cc:302] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab997f5fb85b47258e30650e4dbe3399, e79be34f85714ea19d499cfe703ad1fb; no voters:
I20260619 22:26:34.562050 13650 raft_consensus.cc:3037] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:34.562228 13870 raft_consensus.cc:2781] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:34.562484 13870 raft_consensus.cc:686] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 1 LEADER]: Becoming Leader. State: Replica: e79be34f85714ea19d499cfe703ad1fb, State: Running, Role: LEADER
I20260619 22:26:34.562747 13870 consensus_queue.cc:237] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } }
I20260619 22:26:34.563150 13650 raft_consensus.cc:2454] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79be34f85714ea19d499cfe703ad1fb in term 1.
I20260619 22:26:34.565129 13348 catalog_manager.cc:5495] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e79be34f85714ea19d499cfe703ad1fb (127.5.176.193). New cstate: current_term: 1 leader_uuid: "e79be34f85714ea19d499cfe703ad1fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79be34f85714ea19d499cfe703ad1fb" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 36023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 } health_report { overall_health: UNKNOWN } } }
I20260619 22:26:34.585737 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.581566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.5.176.193:36023)
0619 22:26:34.581844 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.581850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.581862 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:34.581930 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.581931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.581981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.582084 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.582352 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.582905 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.584058 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.584304 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.584383 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.584717 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.585208 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.585326 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.585425 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.585475 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":103,"threads_started":1}
I20260619 22:26:34.596032 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.592626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:41193 (local address 127.5.176.193:33347)
0619 22:26:34.592959 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.592975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.593070 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.593313 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.593316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.593324 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:34.593444 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.593449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.594208 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.594212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.594822 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.594830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.595280 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.595283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.595381 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.595428 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.595730 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.595739 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.595801 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.595880 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":154,"threads_started":1}
I20260619 22:26:34.596015 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.592900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:33347 (local address 127.5.176.195:41193)
0619 22:26:34.593035 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.593039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.593053 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:34.593159 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.593163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.593214 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.593294 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.593555 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.594060 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.594940 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.595171 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.595281 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.595573 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.595642 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.595726 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.595819 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.595870 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:26:34.601614 13790 raft_consensus.cc:1261] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Refusing update from remote peer e79be34f85714ea19d499cfe703ad1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.601614 13650 raft_consensus.cc:1261] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Refusing update from remote peer e79be34f85714ea19d499cfe703ad1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:34.602386 13960 consensus_queue.cc:1035] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab997f5fb85b47258e30650e4dbe3399" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.602566 13870 consensus_queue.cc:1035] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb6ba8a84944f0592643f8bbd4f5c09" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:34.618906 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.615598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:36023 (local address 127.5.176.195:59777)
0619 22:26:34.615721 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:34.615740 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:34.615823 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:34.616171 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:34.616173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:34.616180 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 22:26:34.616266 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.616270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.616925 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.616927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.617560 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:34.617574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.618048 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.618055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:34.618186 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.618217 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.618619 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.618624 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 22:26:34.618702 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 22:26:34.618796 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 22:26:34.618902 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:34.615615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:59777 (local address 127.5.176.193:36023)
0619 22:26:34.615985 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:34.615991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:34.616004 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:34.616044 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:34.616047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:34.616094 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:34.616149 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:34.616339 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.616844 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.617679 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:34.617956 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:34.618054 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:34.618334 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:34.618414 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:34.618495 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:34.618717 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 22:26:34.618767 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260619 22:26:34.641690 13352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:34.641880 13352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:34.644143 13352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8b66f544d7547d3a99984f2ec80bec3: Sending DeleteTablet for 3 replicas of tablet 1c548efe8f7643deb9a99da16d017b9f
I20260619 22:26:34.644802 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c548efe8f7643deb9a99da16d017b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52520
I20260619 22:26:34.644783 13770 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c548efe8f7643deb9a99da16d017b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:43300
I20260619 22:26:34.645020 13935 tablet_replica.cc:330] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: stopping tablet replica
I20260619 22:26:34.645121 13933 tablet_replica.cc:330] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: stopping tablet replica
I20260619 22:26:34.645149 13935 raft_consensus.cc:2229] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:34.645162 13630 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c548efe8f7643deb9a99da16d017b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:34 UTC) from {username='oryx', principal='oryx/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:45942
I20260619 22:26:34.645221 13933 raft_consensus.cc:2229] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:34.645329 13933 raft_consensus.cc:2258] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.645335 13934 tablet_replica.cc:330] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: stopping tablet replica
I20260619 22:26:34.645448 13934 raft_consensus.cc:2229] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:34.645462 13935 raft_consensus.cc:2258] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.645557 13934 raft_consensus.cc:2258] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:34.646206 13935 ts_tablet_manager.cc:1905] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.646332 13934 ts_tablet_manager.cc:1905] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.646617 13933 ts_tablet_manager.cc:1905] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:34.648669 13935 ts_tablet_manager.cc:1918] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:34.648741 13935 log.cc:1196] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/1c548efe8f7643deb9a99da16d017b9f
I20260619 22:26:34.648767 13934 ts_tablet_manager.cc:1918] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:34.648834 13934 log.cc:1196] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/1c548efe8f7643deb9a99da16d017b9f
I20260619 22:26:34.649020 13935 ts_tablet_manager.cc:1939] T 1c548efe8f7643deb9a99da16d017b9f P e79be34f85714ea19d499cfe703ad1fb: Deleting consensus metadata
I20260619 22:26:34.649087 13934 ts_tablet_manager.cc:1939] T 1c548efe8f7643deb9a99da16d017b9f P 5eb6ba8a84944f0592643f8bbd4f5c09: Deleting consensus metadata
I20260619 22:26:34.649775 13339 catalog_manager.cc:4841] TS e79be34f85714ea19d499cfe703ad1fb (127.5.176.193:36023): tablet 1c548efe8f7643deb9a99da16d017b9f (table test-table [id=606c04e2f1894f6bb986a276474878c1]) successfully deleted
I20260619 22:26:34.649947 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 13425
I20260619 22:26:34.651098 13933 ts_tablet_manager.cc:1918] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 22:26:34.651196 13933 log.cc:1196] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/1c548efe8f7643deb9a99da16d017b9f
I20260619 22:26:34.651429 13933 ts_tablet_manager.cc:1939] T 1c548efe8f7643deb9a99da16d017b9f P ab997f5fb85b47258e30650e4dbe3399: Deleting consensus metadata
I20260619 22:26:34.652040 13339 catalog_manager.cc:4841] TS 5eb6ba8a84944f0592643f8bbd4f5c09 (127.5.176.194:43473): tablet 1c548efe8f7643deb9a99da16d017b9f (table test-table [id=606c04e2f1894f6bb986a276474878c1]) successfully deleted
I20260619 22:26:34.652083 13337 catalog_manager.cc:4841] TS ab997f5fb85b47258e30650e4dbe3399 (127.5.176.195:41193): tablet 1c548efe8f7643deb9a99da16d017b9f (table test-table [id=606c04e2f1894f6bb986a276474878c1]) successfully deleted
I20260619 22:26:34.657147 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 13565
I20260619 22:26:34.664170 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 13705
I20260619 22:26:34.671346 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 13321
2026-06-19T22:26:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3456 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13973](info): set up 2 sockets
Jun 19 22:26:34 dist-test-slave-bt27 krb5kdc[13973](info): commencing operation
krb5kdc: starting...
W20260619 22:26:35.722895 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:35 dist-test-slave-bt27 krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:35.844906 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:35035
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:35.954325 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:35.954644 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:35.954717 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:35.958734 13986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:35.958838 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:35.958878 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:35.958940 13986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:35.958978 13986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:35.964597 13986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:35035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:35.965739 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:35.966794 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:35.975145 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:35.975157 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:35.975143 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:35.975427 13986 server_base.cc:1034] running on GCE node
I20260619 22:26:35.976128 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:35.977176 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:35.978334 13986 hybrid_clock.cc:648] HybridClock initialized: now 1781907995978315 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T22:26:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1320 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:36 dist-test-slave-bt27 krb5kdc[13999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:36 dist-test-slave-bt27 krb5kdc[13999](info): set up 2 sockets
Jun 19 22:26:36 dist-test-slave-bt27 krb5kdc[13999](info): commencing operation
krb5kdc: starting...
W20260619 22:26:37.046906 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:37.169845 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:46509
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:37.284518 14012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:37.284848 14012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:37.284925 14012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:37.288987 14012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:37.289093 14012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:37.289134 14012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:37.289193 14012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:37.289234 14012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:37.294679 14012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:46509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:37.295820 14012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:37.296885 14012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:37.305470 14017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:37.305470 14020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:37.305470 14018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:37.305671 14012 server_base.cc:1034] running on GCE node
I20260619 22:26:37.306393 14012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:37.307544 14012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:37.308794 14012 hybrid_clock.cc:648] HybridClock initialized: now 1781907997308773 us; error 33 us; skew 500 ppm
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:37.311630 14012 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:37.312515 14012 webserver.cc:460] Webserver started at http://127.5.176.254:41039/ using document root <none> and password file <none>
I20260619 22:26:37.313031 14012 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:37.313139 14012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:37.313320 14012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:37.315469 14012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "38908046b05443efa5d19c828ff8d821"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "5840f360a19df57fb33111512c816a2e"
server_key_iv: "77dffbb7eee11fb34476cd7afea43da7"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.316046 14012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "38908046b05443efa5d19c828ff8d821"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "5840f360a19df57fb33111512c816a2e"
server_key_iv: "77dffbb7eee11fb34476cd7afea43da7"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.321607 14012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:26:37.324690 14027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.326331 14012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:37.326462 14012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "38908046b05443efa5d19c828ff8d821"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "5840f360a19df57fb33111512c816a2e"
server_key_iv: "77dffbb7eee11fb34476cd7afea43da7"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.326632 14012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:37.348006 14012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:37.351166 14012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:37.351383 14012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:37.358873 14012 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:46509
I20260619 22:26:37.358889 14079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:46509 every 8 connection(s)
I20260619 22:26:37.359881 14012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:37.362793 14080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:37.366551 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14012
I20260619 22:26:37.366652 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:37.366959 5827 external_mini_cluster.cc:1428] Setting key 726ad94a8bb7df55991b3b7b06ab4004
I20260619 22:26:37.369232 14080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Bootstrap starting.
I20260619 22:26:37.371928 14080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:37.372884 14080 log.cc:824] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:37.375340 14080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: No bootstrap required, opened a new log
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:37.379046 14080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:37.379334 14080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:37.379423 14080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38908046b05443efa5d19c828ff8d821, State: Initialized, Role: FOLLOWER
I20260619 22:26:37.379968 14080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:37.380153 14080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:37.380237 14080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:37.380357 14080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:37.381635 14080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:37.382006 14080 leader_election.cc:302] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38908046b05443efa5d19c828ff8d821; no voters:
I20260619 22:26:37.382438 14080 leader_election.cc:288] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:37.382525 14085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:37.382563 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:37.368347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.5.176.254:46509)
0619 22:26:37.368761 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:37.368779 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:37.368829 (+ 50us) server_negotiation.cc:407] Connection header received
0619 22:26:37.369509 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:37.369526 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:37.369858 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:37.370127 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:37.371139 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.372175 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.373389 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.373992 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.374110 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:37.377027 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:37.377032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:37.377045 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:37.377071 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:37.379362 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:37.379896 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:37.379900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:37.379904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:37.380015 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:37.380360 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:37.380363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:37.380365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:37.380816 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:37.380970 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:37.381311 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 22:26:37.381616 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":149,"threads_started":1}
I20260619 22:26:37.382727 14085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 LEADER]: Becoming Leader. State: Replica: 38908046b05443efa5d19c828ff8d821, State: Running, Role: LEADER
I20260619 22:26:37.383054 14085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:37.383818 14080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:37.384299 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38908046b05443efa5d19c828ff8d821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } } }
I20260619 22:26:37.384439 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:37.384907 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38908046b05443efa5d19c828ff8d821. Latest consensus state: current_term: 1 leader_uuid: "38908046b05443efa5d19c828ff8d821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } } }
I20260619 22:26:37.385016 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:37.385622 14089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:37.388577 14089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:37.393663 14089 catalog_manager.cc:1347] Generated new cluster ID: 706451aba28c47da972f10bccf319af4
I20260619 22:26:37.393734 14089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:37.405102 14089 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:37.406432 14089 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:37.410635 14089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Generated new TSK 0
I20260619 22:26:37.411429 14089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:37.445577 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:46509
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:37.559500 14106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:37.559804 14106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:37.559849 14106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:37.564258 14106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:37.564330 14106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:37.564450 14106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:37.570013 14106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:37.571144 14106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:37.572185 14106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:37.581769 14112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:37.581763 14114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:37.581805 14111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:37.582104 14106 server_base.cc:1034] running on GCE node
I20260619 22:26:37.582609 14106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:37.583328 14106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:37.584489 14106 hybrid_clock.cc:648] HybridClock initialized: now 1781907997584471 us; error 32 us; skew 500 ppm
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:37.587827 14106 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:37.588871 14106 webserver.cc:460] Webserver started at http://127.5.176.193:34837/ using document root <none> and password file <none>
I20260619 22:26:37.589475 14106 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:37.589540 14106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:37.589764 14106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:37.592024 14106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "64c0584567da45f6a2e4da0ec404808a"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "1d3aec45a3630387c1c81dc143fa36d4"
server_key_iv: "89d354dc48ef2bab4b8fa5c4a24a8066"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.592630 14106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "64c0584567da45f6a2e4da0ec404808a"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "1d3aec45a3630387c1c81dc143fa36d4"
server_key_iv: "89d354dc48ef2bab4b8fa5c4a24a8066"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.596640 14106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:37.599408 14121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.600759 14106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:37.600867 14106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "64c0584567da45f6a2e4da0ec404808a"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "1d3aec45a3630387c1c81dc143fa36d4"
server_key_iv: "89d354dc48ef2bab4b8fa5c4a24a8066"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.601030 14106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:37.617022 14106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:37.620191 14106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:37.620417 14106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:37.621061 14106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:37.622006 14106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:37.622066 14106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.622143 14106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:37.622205 14106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.631568 14106 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:33419
I20260619 22:26:37.631604 14234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:33419 every 8 connection(s)
I20260619 22:26:37.632490 14106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:37.632855 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14106
I20260619 22:26:37.632999 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:37.633342 5827 external_mini_cluster.cc:1428] Setting key 3710c66f894929adebe237eb69d01cfe
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:37.648996 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:37.634479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:42481 (local address 127.5.176.254:46509)
0619 22:26:37.634621 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:37.634626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:37.635909 (+ 1283us) server_negotiation.cc:407] Connection header received
0619 22:26:37.636830 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:37.636833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:37.636889 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:37.636993 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:37.638545 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.639074 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.641022 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.641649 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.641852 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:37.645221 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:37.645225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:37.645227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:37.645253 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:37.646741 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:37.647273 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:37.647276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:37.647277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:37.647352 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:37.647777 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:37.647780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:37.647781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:37.647965 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:37.648079 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:37.648682 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 22:26:37.648821 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:37.649971 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:37.635022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:42481)
0619 22:26:37.635744 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:37.635788 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:37.636613 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:37.637200 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:37.637207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:37.637584 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:37.638345 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:37.638370 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.639211 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.639218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:37.640854 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:37.640871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.641849 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.641853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:37.642061 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:37.642879 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:37.642904 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:37.644997 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:37.646863 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:37.646870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:37.646889 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:37.647176 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:37.647437 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:37.647439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:37.647440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:37.647687 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:37.648080 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:37.648090 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:37.648428 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 22:26:37.648627 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 22:26:37.648969 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":138,"threads_started":1}
I20260619 22:26:37.651428 14235 heartbeater.cc:347] Connected to a master server at 127.5.176.254:46509
I20260619 22:26:37.651811 14235 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:37.652303 14235 heartbeater.cc:511] Master 127.5.176.254:46509 requested a full tablet report, sending...
I20260619 22:26:37.654065 14044 ts_manager.cc:194] Registered new tserver with Master: 64c0584567da45f6a2e4da0ec404808a (127.5.176.193:33419)
I20260619 22:26:37.655733 14044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:42481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:37.664912 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:46509
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:37.798395 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:37.798651 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:37.798727 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:37.802894 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:37.802954 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:37.803048 14240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:37.808557 14240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:37.809687 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:37.810842 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:37.820668 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:37.820652 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:37.820668 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:37.821158 14240 server_base.cc:1034] running on GCE node
I20260619 22:26:37.821655 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:37.822402 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:37.823587 14240 hybrid_clock.cc:648] HybridClock initialized: now 1781907997823572 us; error 30 us; skew 500 ppm
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:37.826704 14240 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:37.827772 14240 webserver.cc:460] Webserver started at http://127.5.176.194:40233/ using document root <none> and password file <none>
I20260619 22:26:37.828370 14240 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:37.828434 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:37.828650 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:37.830946 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "3a4facef44d84a40a19d1ef718404b54"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "6d7d8a408d8766297c37919c16d27d0c"
server_key_iv: "14dd0fe5d35fa8ea4ca3217f57715de6"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.831566 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "3a4facef44d84a40a19d1ef718404b54"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "6d7d8a408d8766297c37919c16d27d0c"
server_key_iv: "14dd0fe5d35fa8ea4ca3217f57715de6"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.835852 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:37.838449 14255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.839809 14240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:26:37.839937 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "3a4facef44d84a40a19d1ef718404b54"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "6d7d8a408d8766297c37919c16d27d0c"
server_key_iv: "14dd0fe5d35fa8ea4ca3217f57715de6"
server_key_version: "encryptionkey@0"
I20260619 22:26:37.840054 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:37.878036 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:37.881660 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:37.881892 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:37.882557 14240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:37.883340 14240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:37.883389 14240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.883430 14240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:37.883502 14240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:37.893018 14240 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:38461
I20260619 22:26:37.893038 14368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:38461 every 8 connection(s)
I20260619 22:26:37.893920 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:37.902486 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14240
I20260619 22:26:37.902637 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:37.902935 5827 external_mini_cluster.cc:1428] Setting key 4757a06aa7ad4c03561dbbb63cf85726
Jun 19 22:26:37 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907997, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:37.907545 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:37.895856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:54383 (local address 127.5.176.254:46509)
0619 22:26:37.895999 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:37.896002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:37.896992 (+ 990us) server_negotiation.cc:407] Connection header received
0619 22:26:37.897847 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:37.897850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:37.897895 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:37.897977 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:37.899112 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.899645 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.900780 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.900987 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.901066 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:37.903749 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:37.903752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:37.903755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:37.903785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:37.905314 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:37.905854 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:37.905859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:37.905861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:37.905964 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:37.906345 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:37.906350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:37.906353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:37.906592 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:37.906709 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:37.907176 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 22:26:37.907308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:26:37.908211 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:37.896219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:54383)
0619 22:26:37.896839 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:37.896879 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:37.897644 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:37.898168 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:37.898175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:37.898526 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:37.898980 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:37.898993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.899763 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.899766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:37.900651 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:37.900660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:37.901132 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:37.901135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:37.901281 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:37.901869 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:37.901886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:37.903570 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:37.905506 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:37.905511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:37.905522 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:37.905768 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:37.906061 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:37.906063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:37.906064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:37.906256 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:37.906699 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:37.906703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:37.906919 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 22:26:37.907063 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 22:26:37.907307 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":102,"threads_started":1}
I20260619 22:26:37.909303 14369 heartbeater.cc:347] Connected to a master server at 127.5.176.254:46509
I20260619 22:26:37.909564 14369 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:37.910224 14369 heartbeater.cc:511] Master 127.5.176.254:46509 requested a full tablet report, sending...
I20260619 22:26:37.911334 14044 ts_manager.cc:194] Registered new tserver with Master: 3a4facef44d84a40a19d1ef718404b54 (127.5.176.194:38461)
I20260619 22:26:37.911928 14044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:54383
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:37.929515 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:46509
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:38.043283 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:38.043578 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:38.043625 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:38.048013 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:38.048089 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:38.048151 14374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:38.053633 14374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:38.054839 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:38.055931 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:38.065768 14382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.065785 14379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.065894 14374 server_base.cc:1034] running on GCE node
W20260619 22:26:38.065786 14380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.066502 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:38.067168 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:38.068333 14374 hybrid_clock.cc:648] HybridClock initialized: now 1781907998068319 us; error 26 us; skew 500 ppm
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907998, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:38.071321 14374 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:38.072249 14374 webserver.cc:460] Webserver started at http://127.5.176.195:35341/ using document root <none> and password file <none>
I20260619 22:26:38.072837 14374 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:38.072899 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:38.073112 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:38.075223 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "3b5cd20f376440d1a8641b1c2d9dddc3"
format_stamp: "Formatted at 2026-06-19 22:26:38 on dist-test-slave-bt27"
server_key: "6b841781569ba8c6f5679d1ab4dba167"
server_key_iv: "b21df66c57fea4219650099f48ee517f"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.075796 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "3b5cd20f376440d1a8641b1c2d9dddc3"
format_stamp: "Formatted at 2026-06-19 22:26:38 on dist-test-slave-bt27"
server_key: "6b841781569ba8c6f5679d1ab4dba167"
server_key_iv: "b21df66c57fea4219650099f48ee517f"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.079595 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:38.082285 14389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.083619 14374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:38.083815 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "3b5cd20f376440d1a8641b1c2d9dddc3"
format_stamp: "Formatted at 2026-06-19 22:26:38 on dist-test-slave-bt27"
server_key: "6b841781569ba8c6f5679d1ab4dba167"
server_key_iv: "b21df66c57fea4219650099f48ee517f"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.083933 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:38.097995 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:38.101401 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:38.101563 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:38.102102 14374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:38.103022 14374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:38.103080 14374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.103120 14374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:38.103137 14374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.112368 14374 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42733
I20260619 22:26:38.112391 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42733 every 8 connection(s)
I20260619 22:26:38.113312 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:38.116525 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14374
I20260619 22:26:38.116693 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:38.116976 5827 external_mini_cluster.cc:1428] Setting key 41ae3dab7cb182ecdf4db7309ef18b4d
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907998, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:38.127137 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.115180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:56931 (local address 127.5.176.254:46509)
0619 22:26:38.115322 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.115326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.116226 (+ 900us) server_negotiation.cc:407] Connection header received
0619 22:26:38.117724 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.117727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.117786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.117865 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.119143 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.119684 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.120832 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.121060 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.121173 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.123697 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:38.123701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:38.123706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:38.123735 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:38.125017 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:38.125526 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:38.125531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:38.125533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:38.125619 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:38.125966 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:38.125972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:38.125973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:38.126227 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:38.126350 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:38.126850 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 22:26:38.126973 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:26:38.127976 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.115591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:56931)
0619 22:26:38.116087 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.116130 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.117523 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.118045 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.118053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.118481 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.118993 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.119008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.119818 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.119821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.120704 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.120713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.121212 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.121216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.121390 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.122017 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.122033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.123527 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:38.125138 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:38.125144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:38.125159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:38.125420 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:38.125720 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:38.125723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:38.125725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:38.125871 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:38.126347 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:38.126355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:38.126608 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 22:26:38.126798 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 22:26:38.127136 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260619 22:26:38.129184 14503 heartbeater.cc:347] Connected to a master server at 127.5.176.254:46509
I20260619 22:26:38.129465 14503 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:38.130125 14503 heartbeater.cc:511] Master 127.5.176.254:46509 requested a full tablet report, sending...
I20260619 22:26:38.131212 14044 ts_manager.cc:194] Registered new tserver with Master: 3b5cd20f376440d1a8641b1c2d9dddc3 (127.5.176.195:42733)
I20260619 22:26:38.131752 14044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:56931
I20260619 22:26:38.143126 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab.
Entry for principal oryx/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab.
WARNING: no policy specified for oryx/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab.
Entry for principal oryx/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab.
WARNING: no policy specified for oryx/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab.
Entry for principal oryx/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab.
I20260619 22:26:38.214993 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14106
I20260619 22:26:38.220605 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14240
I20260619 22:26:38.226270 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14374
I20260619 22:26:38.231766 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14012
I20260619 22:26:38.238366 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:46509
--webserver_interface=127.5.176.254
--webserver_port=41039
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:38.350553 14512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:38.350854 14512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:38.350898 14512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:38.355046 14512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:38.355122 14512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:38.355142 14512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:38.355159 14512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:38.355175 14512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:38.360755 14512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:46509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:38.361920 14512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:38.362990 14512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:38.371517 14520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.371517 14517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.371512 14518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.371874 14512 server_base.cc:1034] running on GCE node
I20260619 22:26:38.372485 14512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:38.373631 14512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:38.374826 14512 hybrid_clock.cc:648] HybridClock initialized: now 1781907998374813 us; error 41 us; skew 500 ppm
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907998, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:38.377559 14512 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:38.378559 14512 webserver.cc:460] Webserver started at http://127.5.176.254:41039/ using document root <none> and password file <none>
I20260619 22:26:38.379138 14512 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:38.379261 14512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:38.383345 14512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 22:26:38.385706 14527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.387095 14512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:38.387253 14512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "38908046b05443efa5d19c828ff8d821"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "5840f360a19df57fb33111512c816a2e"
server_key_iv: "77dffbb7eee11fb34476cd7afea43da7"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.387840 14512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:38.405884 14512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:38.408991 14512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:38.409225 14512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:38.416445 14512 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:46509
I20260619 22:26:38.416550 14579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:46509 every 8 connection(s)
I20260619 22:26:38.417613 14512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:38.422026 14580 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:26:38.423831 14580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Bootstrap starting.
I20260619 22:26:38.424729 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14512
I20260619 22:26:38.425231 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:33419
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:46509
--webserver_port=34837
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:26:38.441042 14580 log.cc:824] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:38.447475 14580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:26:38.447863 14580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Bootstrap complete.
I20260619 22:26:38.452064 14580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:38.452447 14580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38908046b05443efa5d19c828ff8d821, State: Initialized, Role: FOLLOWER
I20260619 22:26:38.452986 14580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:38.453104 14580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:38.453190 14580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:38.453284 14580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:26:38.455289 14580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:38.455628 14580 leader_election.cc:302] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38908046b05443efa5d19c828ff8d821; no voters:
I20260619 22:26:38.455972 14580 leader_election.cc:288] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:26:38.456060 14584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:26:38.456244 14584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [term 2 LEADER]: Becoming Leader. State: Replica: 38908046b05443efa5d19c828ff8d821, State: Running, Role: LEADER
I20260619 22:26:38.456548 14584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } }
I20260619 22:26:38.457212 14580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:38.458019 14586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38908046b05443efa5d19c828ff8d821. Latest consensus state: current_term: 2 leader_uuid: "38908046b05443efa5d19c828ff8d821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } } }
I20260619 22:26:38.458045 14585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38908046b05443efa5d19c828ff8d821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38908046b05443efa5d19c828ff8d821" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 46509 } } }
I20260619 22:26:38.458176 14586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:38.458199 14585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:38.458657 14589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:38.461961 14589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:38.462615 14589 catalog_manager.cc:1259] Loaded cluster ID: 706451aba28c47da972f10bccf319af4
I20260619 22:26:38.462677 14589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:38.464953 14589 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:38.465617 14589 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38908046b05443efa5d19c828ff8d821: Loaded TSK: 0
I20260619 22:26:38.466810 14589 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:26:38.538388 14582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:38.538681 14582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:38.538761 14582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:38.542805 14582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:38.542893 14582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:38.542985 14582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:38.548282 14582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.193.keytab
--principal=oryx/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:33419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=34837
--tserver_master_addrs=127.5.176.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:38.549302 14582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:38.550351 14582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:38.560902 14606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.560904 14609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.560926 14607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.561196 14582 server_base.cc:1034] running on GCE node
I20260619 22:26:38.561736 14582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:38.562503 14582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:38.563683 14582 hybrid_clock.cc:648] HybridClock initialized: now 1781907998563655 us; error 49 us; skew 500 ppm
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907998, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:38.566509 14582 init.cc:377] Logged in from keytab as oryx/127.5.176.193@KRBTEST.COM (short username oryx)
I20260619 22:26:38.567543 14582 webserver.cc:460] Webserver started at http://127.5.176.193:34837/ using document root <none> and password file <none>
I20260619 22:26:38.568204 14582 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:38.568279 14582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:38.572407 14582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:26:38.574628 14616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.575945 14582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:38.576040 14582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "64c0584567da45f6a2e4da0ec404808a"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "1d3aec45a3630387c1c81dc143fa36d4"
server_key_iv: "89d354dc48ef2bab4b8fa5c4a24a8066"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.576634 14582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:38.591667 14582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:38.595081 14582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:38.595289 14582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:38.595956 14582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:38.596827 14582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:38.596872 14582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.596932 14582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:38.596963 14582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.606499 14582 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:33419
I20260619 22:26:38.607710 14582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:38.608783 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:33419 every 8 connection(s)
I20260619 22:26:38.612123 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14582
I20260619 22:26:38.612741 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:38461
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:46509
--webserver_port=40233
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:26:38.625927 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.610793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55031 (local address 127.5.176.254:46509)
0619 22:26:38.614430 (+ 3637us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.614449 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.614495 (+ 46us) server_negotiation.cc:407] Connection header received
0619 22:26:38.614761 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.614791 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.615210 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.615565 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.617004 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.618833 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.620465 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.621375 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.621582 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.625231 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:55031: BlockingRecv error: recv got EOF from 127.5.176.193:55031 (error 108)
Metrics: {"server-negotiator.queue_time_us":3453,"thread_start_us":106,"threads_started":1}
I20260619 22:26:38.625962 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.611045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:55031)
0619 22:26:38.611874 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.611930 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.613244 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.615636 (+ 2392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.615644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.616190 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.616799 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.616818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.619000 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.619003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.620322 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.620335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.621590 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.621593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.621740 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.622432 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.622449 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.625002 (+ 2553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":153,"threads_started":1}
W20260619 22:26:38.626633 14730 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:46509 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.176.254:46509: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:38.632148 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.627096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:48467)
0619 22:26:38.627221 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.627237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.627325 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.627580 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.627583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.627745 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.627834 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.627839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.628561 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.628566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.629103 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.629110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.629553 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.629556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.629645 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.630220 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.630232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.632003 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 22:26:38.632227 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.627178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:48467 (local address 127.5.176.254:46509)
0619 22:26:38.627305 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.627309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.627322 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:38.627429 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.627432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.627481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.627566 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.627945 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.628456 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.629221 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.629442 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.629549 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.632115 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:48467: BlockingRecv error: recv got EOF from 127.5.176.193:48467 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:38.637507 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.632708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:35801)
0619 22:26:38.632794 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.632809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.632882 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.633088 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.633090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.633203 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.633267 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.633272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.634234 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.634236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.634733 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.634740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.635266 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.635269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.635329 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.635762 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.635772 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.637415 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 22:26:38.637678 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.632762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:35801 (local address 127.5.176.254:46509)
0619 22:26:38.632866 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.632870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.632884 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:38.632946 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.632949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.632988 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.633069 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.633374 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.634124 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.634834 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.635164 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.635262 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.637562 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:35801: BlockingRecv error: recv got EOF from 127.5.176.193:35801 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 22:26:38.638010 14730 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:26:38.728652 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:38.728916 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:38.728961 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:38.732964 14734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:38.733018 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:38.733075 14734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:38.738175 14734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.194.keytab
--principal=oryx/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:38461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=40233
--tserver_master_addrs=127.5.176.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:38.739230 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:38.740233 14734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:38.749759 14742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.749763 14739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.749781 14740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.749922 14734 server_base.cc:1034] running on GCE node
I20260619 22:26:38.750497 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:38.751155 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:38.752323 14734 hybrid_clock.cc:648] HybridClock initialized: now 1781907998752308 us; error 28 us; skew 500 ppm
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907998, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:38.755206 14734 init.cc:377] Logged in from keytab as oryx/127.5.176.194@KRBTEST.COM (short username oryx)
I20260619 22:26:38.756278 14734 webserver.cc:460] Webserver started at http://127.5.176.194:40233/ using document root <none> and password file <none>
I20260619 22:26:38.756808 14734 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:38.756877 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:38.760857 14734 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:26:38.762848 14749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.764068 14734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:38.764158 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "3a4facef44d84a40a19d1ef718404b54"
format_stamp: "Formatted at 2026-06-19 22:26:37 on dist-test-slave-bt27"
server_key: "6d7d8a408d8766297c37919c16d27d0c"
server_key_iv: "14dd0fe5d35fa8ea4ca3217f57715de6"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.764694 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:38.780505 14734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:38.783893 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:38.784051 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:38.784596 14734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:38.785380 14734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:38.785425 14734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.785465 14734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:38.785485 14734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.794878 14734 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:38461
I20260619 22:26:38.794896 14862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:38461 every 8 connection(s)
I20260619 22:26:38.795887 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:38.799763 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14734
I20260619 22:26:38.800246 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:42733
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:46509
--webserver_port=35341
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:26:38.807574 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.797641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:37245 (local address 127.5.176.254:46509)
0619 22:26:38.797775 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.797778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.798791 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 22:26:38.799752 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.799754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.799805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.799913 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.801509 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.802388 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.803496 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.803909 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.804031 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.807397 (+ 3366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:37245: BlockingRecv error: recv got EOF from 127.5.176.194:37245 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:26:38.808004 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.798030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:37245)
0619 22:26:38.798651 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.798692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.799523 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.800122 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.800131 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.800646 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.801347 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.801368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.802504 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.802507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.803373 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.803381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.804091 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.804095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.804239 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.804871 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.804887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.807184 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":126,"threads_started":1}
W20260619 22:26:38.808588 14863 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:46509 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.176.254:46509: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:38.813927 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.809050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:52209)
0619 22:26:38.809161 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.809174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.809248 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.809478 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.809481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.809637 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.809741 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.809746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.810525 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.810531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.811070 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.811077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.811508 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.811511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.811600 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.812074 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.812086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.813770 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 22:26:38.814002 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.809147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52209 (local address 127.5.176.254:46509)
0619 22:26:38.809251 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.809255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.809269 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:38.809333 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.809336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.809380 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.809465 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.809855 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.810425 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.811183 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.811403 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.811510 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.813888 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:52209: BlockingRecv error: recv got EOF from 127.5.176.194:52209 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:38.819952 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.814670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:48023)
0619 22:26:38.814895 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.814910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.815017 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.815258 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.815261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.815424 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.815493 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.815498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.816217 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.816223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.816746 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.816754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.817187 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.817191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.817268 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.817887 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.817900 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.819794 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165}
W20260619 22:26:38.820163 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.814781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:48023 (local address 127.5.176.254:46509)
0619 22:26:38.814881 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.814885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.814977 (+ 92us) server_negotiation.cc:407] Connection header received
0619 22:26:38.815116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.815121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.815162 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.815245 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.815607 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.816112 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.816871 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.817080 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.817188 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.819927 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:48023: BlockingRecv error: recv got EOF from 127.5.176.194:48023 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 22:26:38.820411 14863 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:26:38.917152 14866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:38.917456 14866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:38.917505 14866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:38.921618 14866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:38.921705 14866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:38.921767 14866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:38.927107 14866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/krb5kdc/oryx_127.5.176.195.keytab
--principal=oryx/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:42733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=35341
--tserver_master_addrs=127.5.176.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:38.928148 14866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:38.929137 14866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:38.938772 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:38.938778 14874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.938870 14866 server_base.cc:1034] running on GCE node
W20260619 22:26:38.938784 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:38.939427 14866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:38.940099 14866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:38.941300 14866 hybrid_clock.cc:648] HybridClock initialized: now 1781907998941285 us; error 27 us; skew 500 ppm
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781907998, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:38.944206 14866 init.cc:377] Logged in from keytab as oryx/127.5.176.195@KRBTEST.COM (short username oryx)
I20260619 22:26:38.945111 14866 webserver.cc:460] Webserver started at http://127.5.176.195:35341/ using document root <none> and password file <none>
I20260619 22:26:38.945676 14866 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:38.945796 14866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:38.949548 14866 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:38.951640 14881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.952854 14866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:38.953001 14866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "3b5cd20f376440d1a8641b1c2d9dddc3"
format_stamp: "Formatted at 2026-06-19 22:26:38 on dist-test-slave-bt27"
server_key: "6b841781569ba8c6f5679d1ab4dba167"
server_key_iv: "b21df66c57fea4219650099f48ee517f"
server_key_version: "encryptionkey@0"
I20260619 22:26:38.953557 14866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:38.972944 14866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:38.976313 14866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:38.976521 14866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:38.977115 14866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:38.977938 14866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:38.978012 14866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.978087 14866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:38.978121 14866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:38.987068 14866 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42733
I20260619 22:26:38.987095 14994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42733 every 8 connection(s)
I20260619 22:26:38.988096 14866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:38.997617 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 14866
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:38 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 22:26:38.999615 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:38.989962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:54473 (local address 127.5.176.254:46509)
0619 22:26:38.990117 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.990120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:38.991332 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 22:26:38.992353 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:38.992357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:38.992421 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:38.992539 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:38.993779 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.994379 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.995563 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.995794 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.995918 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.999494 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:54473: BlockingRecv error: recv got EOF from 127.5.176.195:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 22:26:39.000250 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.990399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:54473)
0619 22:26:38.991151 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:38.991216 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:38.992127 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:38.992734 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:38.992741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:38.993130 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:38.993612 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.993626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.994514 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.994520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.995413 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:38.995423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:38.995966 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:38.995975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:38.996155 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:38.996831 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:38.996849 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:38.999389 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":110,"threads_started":1}
W20260619 22:26:39.000753 14995 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:46509 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.176.254:46509: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:39.006242 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:39.001200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:39047)
0619 22:26:39.001321 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:39.001335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:39.001403 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:39.001615 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:39.001617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:39.001776 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:39.001876 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.001882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.002615 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.002621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.003165 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.003173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.003592 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.003596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.003682 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.004047 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:39.004058 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:39.006075 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 22:26:39.006333 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:39.001245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:39047 (local address 127.5.176.254:46509)
0619 22:26:39.001343 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:39.001352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:39.001365 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:39.001483 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:39.001485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:39.001515 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:39.001575 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:39.001976 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.002512 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.003279 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.003487 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.003590 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.006212 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:39047: BlockingRecv error: recv got EOF from 127.5.176.195:39047 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:26:39.009126 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:38.998714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.5.176.254:46509)
0619 22:26:38.998938 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:38.998941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:39.000038 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 22:26:39.000197 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:39.000200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:39.000250 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:39.000363 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:39.001200 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.002030 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.002895 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.003131 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.003240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.004758 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:39.004764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:39.004779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:39.004812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:39.007048 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:39.007504 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:39.007511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:39.007518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:39.007645 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:39.007934 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:39.007938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:39.007940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:39.008300 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:39.008503 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:39.008778 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 22:26:39.009014 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:39.011441 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:39.006838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:60115)
0619 22:26:39.006928 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:39.006943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:39.007040 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:39.007263 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:39.007266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:39.007414 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:39.007479 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.007484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.008163 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.008169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.008673 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.008681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.009196 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.009199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.009284 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.009652 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:39.009662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:39.011323 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 22:26:39.011623 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:39.006923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:60115 (local address 127.5.176.254:46509)
0619 22:26:39.007072 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:39.007078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:39.007093 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:26:39.007125 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:39.007128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:39.007160 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:39.007249 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:39.007584 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.008064 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.008783 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.009091 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.009194 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.011438 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:60115: BlockingRecv error: recv got EOF from 127.5.176.195:60115 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 22:26:39.011897 14995 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:39.646288 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:39.638697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:55363)
0619 22:26:39.639102 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:39.639142 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:39.639319 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:39.639664 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:39.639694 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:39.639910 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:39.640010 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.640016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.641221 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.641227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.642349 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.642361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.642957 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.642961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.643046 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.643678 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:39.643696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:39.646088 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":90,"threads_started":1}
W20260619 22:26:39.646425 15001 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:39.639054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55363 (local address 127.5.176.254:46509)
0619 22:26:39.639345 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:39.639352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:39.639371 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:26:39.639427 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:39.639431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:39.639492 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:39.639602 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:39.640132 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.641057 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.642479 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.642838 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.642956 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.646280 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:55363: BlockingRecv error: recv got EOF from 127.5.176.193:55363 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:39 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:39.826912 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:39.821144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:45503)
0619 22:26:39.821472 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:39.821486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:39.821588 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:39.821859 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:39.821862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:39.822056 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:39.822257 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.822266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.823024 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.823026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.823608 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:39.823616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.824068 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.824071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:39.824131 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.824646 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:39.824657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:39.826755 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
W20260619 22:26:39.827216 15001 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:39.821237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:45503 (local address 127.5.176.254:46509)
0619 22:26:39.821395 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:39.821399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:39.821603 (+ 204us) server_negotiation.cc:407] Connection header received
0619 22:26:39.821678 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:39.821681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:39.821755 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:39.821825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:39.822381 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.822926 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.823729 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:39.823962 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:39.824051 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:39.827064 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:45503: BlockingRecv error: recv got EOF from 127.5.176.194:45503 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 22:26:40 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:40 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:40.021697 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:40.012811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:51669)
0619 22:26:40.013100 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:40.013114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:40.013227 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:40.013725 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:40.013730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:40.013990 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:40.014205 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:40.014214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.015697 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.015702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:40.016710 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:40.016729 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.017620 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.017627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:40.017723 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:40.018359 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:40.018371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:40.021350 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
W20260619 22:26:40.022012 15001 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:40.013001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:51669 (local address 127.5.176.254:46509)
0619 22:26:40.013195 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:40.013199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:40.013216 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:26:40.013379 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:40.013382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:40.013436 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:40.013526 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:40.014469 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.015417 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.017094 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.017456 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.017581 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:40.021697 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:51669: BlockingRecv error: recv got EOF from 127.5.176.195:51669 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 22:26:40 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:40 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:40.653131 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:40.647364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:39491)
0619 22:26:40.647648 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:40.647661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:40.647754 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:40.648001 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:40.648003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:40.648179 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:40.648272 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:40.648276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.649214 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.649217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:40.649946 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:40.649955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.650489 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.650491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:40.650601 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:40.651119 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:40.651130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:40.652932 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
W20260619 22:26:40.653182 15005 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:40.647492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:39491 (local address 127.5.176.254:46509)
0619 22:26:40.647749 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:40.647752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:40.647766 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:40.647831 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:40.647833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:40.647877 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:40.647963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:40.648413 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.649089 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.650064 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.650341 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.650465 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:40.653059 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:39491: BlockingRecv error: recv got EOF from 127.5.176.193:39491 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
Jun 19 22:26:40 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:40 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:40.833431 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:40.827816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:45695)
0619 22:26:40.828153 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:40.828167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:40.828305 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:40.828591 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:40.828593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:40.828789 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:40.828884 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:40.828890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.829663 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.829666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:40.830265 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:40.830274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.830784 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.830787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:40.830853 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:40.831390 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:40.831402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:40.833289 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":77,"threads_started":1}
W20260619 22:26:40.833707 15005 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:40.827937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:45695 (local address 127.5.176.254:46509)
0619 22:26:40.828080 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:40.828087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:40.828259 (+ 172us) server_negotiation.cc:407] Connection header received
0619 22:26:40.828406 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:40.828408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:40.828466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:40.828535 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:40.829012 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.829540 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.830420 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:40.830655 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:40.830765 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:40.833473 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:45695: BlockingRecv error: recv got EOF from 127.5.176.194:45695 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 22:26:41 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:41 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:41.031085 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:41.022920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:34871)
0619 22:26:41.023209 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:41.023223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:41.023318 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:41.023574 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:41.023576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:41.023752 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:41.023854 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:41.023859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.024615 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.024618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:41.025163 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:41.025170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.025599 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.025601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:41.025663 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:41.028999 (+ 3336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:41.029011 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:41.030930 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
W20260619 22:26:41.031169 15005 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:41.023094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:34871 (local address 127.5.176.254:46509)
0619 22:26:41.023285 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:41.023289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:41.023304 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:26:41.023419 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:41.023421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:41.023466 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:41.023541 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:41.023963 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.024514 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.025283 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.025493 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.025584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:41.031075 (+ 5491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:34871: BlockingRecv error: recv got EOF from 127.5.176.195:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 22:26:41 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:41 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:41.659557 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:41.653967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:41827)
0619 22:26:41.654262 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:41.654277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:41.654365 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:41.654565 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:41.654566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:41.654736 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:41.654838 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:41.654843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.655574 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.655577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:41.656137 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:41.656145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.656606 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.656612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:41.656739 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:41.657313 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:41.657325 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:41.659435 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
W20260619 22:26:41.659809 15009 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:41.654041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:41827 (local address 127.5.176.254:46509)
0619 22:26:41.654304 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:41.654308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:41.654322 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:41.654433 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:41.654435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:41.654476 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:41.654550 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:41.654948 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.655470 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.656255 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.656473 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.656612 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:41.659658 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:41827: BlockingRecv error: recv got EOF from 127.5.176.193:41827 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":118,"threads_started":1}
Jun 19 22:26:41 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:41 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:41.839926 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:41.834385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:34205)
0619 22:26:41.834662 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:41.834678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:41.834785 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:41.835013 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:41.835015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:41.835186 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:41.835278 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:41.835283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.836120 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.836123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:41.836663 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:41.836670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.837152 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.837155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:41.837226 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:41.837810 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:41.837822 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:41.839753 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
W20260619 22:26:41.840065 15009 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:41.834460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:34205 (local address 127.5.176.254:46509)
0619 22:26:41.834624 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:41.834628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:41.834739 (+ 111us) server_negotiation.cc:407] Connection header received
0619 22:26:41.834894 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:41.834896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:41.834940 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:41.835003 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:41.835427 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.835996 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.836791 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:41.837010 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:41.837101 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:41.839927 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:34205: BlockingRecv error: recv got EOF from 127.5.176.194:34205 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 22:26:42 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:42 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:42.037691 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:42.031924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:36911)
0619 22:26:42.032278 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:42.032292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:42.032427 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:42.032705 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:42.032708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:42.032921 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:42.033011 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:42.033016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.033808 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.033813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:42.034412 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:42.034421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.034897 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.034900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:42.034986 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:42.035575 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:42.035586 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:42.037516 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":98,"threads_started":1}
W20260619 22:26:42.037792 15009 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:42.032045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:36911 (local address 127.5.176.254:46509)
0619 22:26:42.032221 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:42.032226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:42.032368 (+ 142us) server_negotiation.cc:407] Connection header received
0619 22:26:42.032529 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:42.032533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:42.032584 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:42.032688 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:42.033129 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.033693 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.034534 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.034779 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.034891 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:42.037648 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:36911: BlockingRecv error: recv got EOF from 127.5.176.195:36911 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 22:26:42 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:42 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:42.665987 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:42.660331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:54473)
0619 22:26:42.660637 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:42.660652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:42.660746 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:42.661120 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:42.661123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:42.661317 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:42.661418 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:42.661424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.662335 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.662339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:42.662904 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:42.662912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.663357 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.663359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:42.663420 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:42.663954 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:42.663965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:42.665864 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
W20260619 22:26:42.666177 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:42.660456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:54473 (local address 127.5.176.254:46509)
0619 22:26:42.660688 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:42.660691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:42.660705 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:42.660888 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:42.660891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:42.660944 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:42.661058 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:42.661552 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.662165 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.663021 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.663250 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.663342 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:42.666014 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:54473: BlockingRecv error: recv got EOF from 127.5.176.193:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":57,"threads_started":1}
Jun 19 22:26:42 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:42 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:42.846336 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:42.840840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:45985)
0619 22:26:42.841126 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:42.841139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:42.841249 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:42.841528 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:42.841531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:42.841713 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:42.841814 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:42.841820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.842616 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.842619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:42.843162 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:42.843170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.843671 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.843675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:42.843773 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:42.844337 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:42.844349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:42.846142 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":113,"threads_started":1}
W20260619 22:26:42.846398 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:42.840940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:45985 (local address 127.5.176.254:46509)
0619 22:26:42.841085 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:42.841089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:42.841190 (+ 101us) server_negotiation.cc:407] Connection header received
0619 22:26:42.841337 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:42.841339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:42.841383 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:42.841446 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:42.841925 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.842513 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.843273 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:42.843517 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:42.843598 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:42.846290 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:45985: BlockingRecv error: recv got EOF from 127.5.176.194:45985 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 22:26:43 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:43 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:43.044678 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:43.038592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:54139)
0619 22:26:43.039094 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:43.039107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:43.039244 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:43.039526 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:43.039529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:43.039730 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:43.039837 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:43.039842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.040610 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.040617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:43.041162 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:43.041169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.041621 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.041625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:43.041710 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:43.042390 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:43.042405 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:43.044508 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":86,"threads_started":1}
W20260619 22:26:43.044739 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:43.038780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:54139 (local address 127.5.176.254:46509)
0619 22:26:43.039044 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:43.039049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:43.039182 (+ 133us) server_negotiation.cc:407] Connection header received
0619 22:26:43.039347 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:43.039350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:43.039405 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:43.039497 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:43.039954 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.040497 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.041278 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.041512 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.041620 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:43.044629 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:54139: BlockingRecv error: recv got EOF from 127.5.176.195:54139 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
Jun 19 22:26:43 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:43 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:43.672233 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:43.666900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:48153)
0619 22:26:43.667177 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:43.667192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:43.667281 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:43.667566 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:43.667568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:43.667748 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:43.667842 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:43.667848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.668663 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.668666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:43.669305 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:43.669314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.669809 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.669812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:43.669883 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:43.670382 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:43.670393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:43.672117 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260619 22:26:43.672415 15017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:43.666973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:48153 (local address 127.5.176.254:46509)
0619 22:26:43.667191 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:43.667194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:43.667248 (+ 54us) server_negotiation.cc:407] Connection header received
0619 22:26:43.667349 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:43.667351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:43.667407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:43.667513 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:43.668013 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.668554 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.669449 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.669704 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.669796 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:43.672248 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:48153: BlockingRecv error: recv got EOF from 127.5.176.193:48153 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
Jun 19 22:26:43 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:43 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:43.852952 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:43.847254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:33441)
0619 22:26:43.847598 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:43.847614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:43.847725 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:43.848026 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:43.848029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:43.848267 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:43.848363 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:43.848369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.849147 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.849153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:43.849724 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:43.849732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.850254 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.850258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:43.850351 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:43.850909 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:43.850930 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:43.852763 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
W20260619 22:26:43.852988 15017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:43.847469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:33441 (local address 127.5.176.254:46509)
0619 22:26:43.847604 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:43.847610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:43.847672 (+ 62us) server_negotiation.cc:407] Connection header received
0619 22:26:43.847848 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:43.847851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:43.847902 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:43.847999 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:43.848490 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.849035 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.849851 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:43.850094 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:43.850252 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:43.852892 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:33441: BlockingRecv error: recv got EOF from 127.5.176.194:33441 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 22:26:44 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:44 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:44.051124 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:44.045574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:51077)
0619 22:26:44.045945 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:44.045958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:44.046086 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:44.046402 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:44.046405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:44.046617 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:44.046707 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:44.046721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.047492 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.047499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:44.048056 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:44.048063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.048513 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.048517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:44.048600 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:44.049092 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:44.049104 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:44.050950 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":86,"threads_started":1}
W20260619 22:26:44.051180 15017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:44.045672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:51077 (local address 127.5.176.254:46509)
0619 22:26:44.045851 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:44.045856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:44.046039 (+ 183us) server_negotiation.cc:407] Connection header received
0619 22:26:44.046230 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:44.046233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:44.046287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:44.046379 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:44.046836 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.047384 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.048177 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.048407 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.048515 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:44.051073 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:51077: BlockingRecv error: recv got EOF from 127.5.176.195:51077 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 22:26:44 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:44 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:44.678537 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:44.673055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:37889)
0619 22:26:44.673326 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:44.673339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:44.673430 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:44.673680 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:44.673682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:44.673869 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:44.673960 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:44.673965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.674820 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.674823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:44.675372 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:44.675380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.675794 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.675796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:44.675866 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:44.676361 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:44.676372 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:44.678365 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
W20260619 22:26:44.678606 15021 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:44.673179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:37889 (local address 127.5.176.254:46509)
0619 22:26:44.673459 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:44.673463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:44.673475 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:26:44.673514 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:44.673516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:44.673558 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:44.673646 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:44.674081 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.674689 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.675468 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.675683 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.675777 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:44.678486 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:37889: BlockingRecv error: recv got EOF from 127.5.176.193:37889 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
Jun 19 22:26:44 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:44 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:44.859815 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:44.853875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:58251)
0619 22:26:44.854195 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:44.854211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:44.854362 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:44.854641 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:44.854645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:44.854890 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:44.854981 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:44.854987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.855757 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.855762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:44.856318 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:44.856326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.856797 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.856801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:44.856890 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:44.857536 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:44.857549 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:44.859688 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
W20260619 22:26:44.859925 15021 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:44.853929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:58251 (local address 127.5.176.254:46509)
0619 22:26:44.854165 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:44.854170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:44.854298 (+ 128us) server_negotiation.cc:407] Connection header received
0619 22:26:44.854474 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:44.854477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:44.854533 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:44.854624 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:44.855098 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.855643 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.856443 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:44.856677 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:44.856785 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:44.859832 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:58251: BlockingRecv error: recv got EOF from 127.5.176.194:58251 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 19 22:26:45 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:45 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:45.057932 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:45.051976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:33477)
0619 22:26:45.052441 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:45.052454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:45.052605 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:45.052969 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:45.052974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:45.053207 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:45.053350 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:45.053358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.054278 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.054281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:45.054830 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:45.054838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.055307 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.055310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:45.055375 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:45.055885 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:45.055896 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:45.057813 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":88,"threads_started":1}
W20260619 22:26:45.058130 15021 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:45.052041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:33477 (local address 127.5.176.254:46509)
0619 22:26:45.052222 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:45.052227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:45.052592 (+ 365us) server_negotiation.cc:407] Connection header received
0619 22:26:45.052770 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:45.052774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:45.052836 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:45.052952 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:45.053502 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.054137 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.054946 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.055169 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.055254 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:45.057977 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:33477: BlockingRecv error: recv got EOF from 127.5.176.195:33477 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 22:26:45 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:45 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:45.684556 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:45.679280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:48937)
0619 22:26:45.679588 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:45.679601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:45.679694 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:45.679934 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:45.679936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:45.680116 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:45.680207 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:45.680213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.681017 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.681020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:45.681572 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:45.681580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.681989 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.681995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:45.682055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:45.682577 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:45.682589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:45.684441 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":69,"threads_started":1}
W20260619 22:26:45.684716 15025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:45.679366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:48937 (local address 127.5.176.254:46509)
0619 22:26:45.679629 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:45.679633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:45.679648 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:26:45.679778 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:45.679780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:45.679822 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:45.679925 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:45.680343 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.680919 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.681672 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.681889 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.681965 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:45.684579 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:48937: BlockingRecv error: recv got EOF from 127.5.176.193:48937 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
Jun 19 22:26:45 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:45 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:45.866461 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:45.860668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:52797)
0619 22:26:45.860979 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:45.861000 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:45.861102 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:45.861360 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:45.861362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:45.861544 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:45.861641 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:45.861646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.862454 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.862456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:45.863013 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:45.863021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.863470 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.863472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:45.863532 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:45.864193 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:45.864204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:45.866274 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
W20260619 22:26:45.866509 15025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:45.860796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52797 (local address 127.5.176.254:46509)
0619 22:26:45.860946 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:45.860949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:45.861098 (+ 149us) server_negotiation.cc:407] Connection header received
0619 22:26:45.861177 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:45.861179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:45.861242 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:45.861325 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:45.861756 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.862343 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.863130 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:45.863365 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:45.863456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:45.866403 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:52797: BlockingRecv error: recv got EOF from 127.5.176.194:52797 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 22:26:46 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:46 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:46.068557 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:46.058775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:51423)
0619 22:26:46.059132 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:46.059145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:46.059287 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:46.059580 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:46.059583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:46.059803 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:46.059892 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:46.059898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.060656 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.060662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:46.061215 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:46.061222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.061690 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.061694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:46.061778 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:46.066413 (+ 4635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:46.066427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:46.068420 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":108,"threads_started":1}
W20260619 22:26:46.068768 15025 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:46.058825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:51423 (local address 127.5.176.254:46509)
0619 22:26:46.059014 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:46.059019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:46.059225 (+ 206us) server_negotiation.cc:407] Connection header received
0619 22:26:46.059411 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:46.059414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:46.059471 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:46.059558 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:46.060013 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.060547 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.061351 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.061578 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.061687 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:46.068642 (+ 6955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:51423: BlockingRecv error: recv got EOF from 127.5.176.195:51423 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 22:26:46 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:46 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:46.691658 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:46.685508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:42263)
0619 22:26:46.685777 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:46.685790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:46.685899 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:46.686334 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:46.686336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:46.686514 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:46.686614 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:46.686620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.687421 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.687424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:46.687975 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:46.687983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.688454 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.688457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:46.688521 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:46.689088 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:46.689101 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:46.691516 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
W20260619 22:26:46.691859 15029 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:46.685760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:42263 (local address 127.5.176.254:46509)
0619 22:26:46.686027 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:46.686031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:46.686047 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:26:46.686091 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:46.686094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:46.686179 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:46.686291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:46.686754 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.687322 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.688084 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.688339 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.688430 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:46.691715 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:42263: BlockingRecv error: recv got EOF from 127.5.176.193:42263 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
Jun 19 22:26:46 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:46 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:46.873805 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:46.867502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:52159)
0619 22:26:46.867887 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:46.867902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:46.868053 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:46.868418 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:46.868423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:46.868621 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:46.868769 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:46.868778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.869872 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.869875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:46.870493 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:46.870502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.871020 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.871023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:46.871091 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:46.871677 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:46.871697 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:46.873684 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":90,"threads_started":1}
W20260619 22:26:46.874210 15029 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:46.867631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52159 (local address 127.5.176.254:46509)
0619 22:26:46.867787 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:46.867793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:46.868009 (+ 216us) server_negotiation.cc:407] Connection header received
0619 22:26:46.868193 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:46.868199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:46.868284 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:46.868376 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:46.868901 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.869748 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.870627 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:46.870889 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:46.870983 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:46.873888 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:52159: BlockingRecv error: recv got EOF from 127.5.176.194:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 22:26:47 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:47 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:47.074474 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:47.069303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:40573)
0619 22:26:47.069549 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:47.069563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:47.069661 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:47.069893 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:47.069896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:47.070072 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:47.070227 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:47.070234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.070960 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.070963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:47.071510 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:47.071517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.071899 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.071901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:47.071961 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:47.072443 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:47.072454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:47.074351 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
W20260619 22:26:47.074607 15029 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:47.069354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:40573 (local address 127.5.176.254:46509)
0619 22:26:47.069490 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:47.069493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:47.069630 (+ 137us) server_negotiation.cc:407] Connection header received
0619 22:26:47.069727 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:47.069729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:47.069776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:47.069846 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:47.070346 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.070872 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.071603 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.071808 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.071881 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:47.074486 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:40573: BlockingRecv error: recv got EOF from 127.5.176.195:40573 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 22:26:47 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:47 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:47.698387 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:47.692667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:55897)
0619 22:26:47.692938 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:47.692952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:47.693042 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:47.693345 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:47.693349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:47.693581 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:47.693680 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:47.693687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.694656 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.694660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:47.695301 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:47.695311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.695725 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.695727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:47.695789 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:47.696292 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:47.696303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:47.698251 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
W20260619 22:26:47.698671 15033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:47.692815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55897 (local address 127.5.176.254:46509)
0619 22:26:47.693044 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:47.693048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:47.693063 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:26:47.693105 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:47.693107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:47.693148 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:47.693256 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:47.693804 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.694531 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.695409 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.695624 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.695703 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:47.698386 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:55897: BlockingRecv error: recv got EOF from 127.5.176.193:55897 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
Jun 19 22:26:47 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:47 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:47.879735 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:47.874596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:32873)
0619 22:26:47.874859 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:47.874873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:47.874969 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:47.875201 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:47.875203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:47.875381 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:47.875475 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:47.875481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.876166 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.876168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:47.876713 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:47.876720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.877107 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.877109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:47.877173 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:47.877733 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:47.877745 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:47.879599 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
W20260619 22:26:47.879947 15033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:47.874730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:32873 (local address 127.5.176.254:46509)
0619 22:26:47.874889 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:47.874893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:47.874921 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:26:47.875055 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:47.875057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:47.875099 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:47.875166 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:47.875572 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.876080 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.876807 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:47.877011 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:47.877084 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:47.879761 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:32873: BlockingRecv error: recv got EOF from 127.5.176.194:32873 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 22:26:48 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:48 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:48.081027 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:48.075257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:52677)
0619 22:26:48.075542 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:48.075555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:48.075656 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:48.075930 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:48.075932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:48.076104 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:48.076209 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:48.076215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.076993 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.076996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:48.077569 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:48.077578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.078066 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.078069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:48.078281 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:48.078837 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:48.078849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:48.080812 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
W20260619 22:26:48.081072 15033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:48.075355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:52677 (local address 127.5.176.254:46509)
0619 22:26:48.075545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:48.075550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:48.075634 (+ 84us) server_negotiation.cc:407] Connection header received
0619 22:26:48.075768 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:48.075770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:48.075819 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:48.075899 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:48.076342 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.076890 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.077681 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.077935 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.078016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:48.080974 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:52677: BlockingRecv error: recv got EOF from 127.5.176.195:52677 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 22:26:48 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:48 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:48.704610 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:48.699232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:38733)
0619 22:26:48.699530 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:48.699544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:48.699631 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:48.699873 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:48.699875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:48.700048 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:48.700140 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:48.700145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.700875 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.700877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:48.701428 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:48.701436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.701915 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.701918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:48.701983 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:48.702526 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:48.702538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:48.704432 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
W20260619 22:26:48.704726 15037 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:48.699322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:38733 (local address 127.5.176.254:46509)
0619 22:26:48.699619 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:48.699623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:48.699636 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:48.699700 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:48.699702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:48.699766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:48.699856 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:48.700239 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.700776 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.701556 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.701793 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.701894 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:48.704557 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:38733: BlockingRecv error: recv got EOF from 127.5.176.193:38733 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
Jun 19 22:26:48 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:48 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:48.886484 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:48.880704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:56063)
0619 22:26:48.881024 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:48.881037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:48.881169 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:48.881457 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:48.881460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:48.881674 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:48.881787 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:48.881792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.882603 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.882608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:48.883171 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:48.883178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.883628 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.883632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:48.883728 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:48.884291 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:48.884312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:48.886310 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
W20260619 22:26:48.886543 15037 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:48.880786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:56063 (local address 127.5.176.254:46509)
0619 22:26:48.880944 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:48.880949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:48.881119 (+ 170us) server_negotiation.cc:407] Connection header received
0619 22:26:48.881287 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:48.881290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:48.881342 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:48.881438 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:48.881909 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.882493 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.883291 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:48.883514 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:48.883627 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:48.886435 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:56063: BlockingRecv error: recv got EOF from 127.5.176.194:56063 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 22:26:49 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:49 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:49.087383 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:49.082016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:44291)
0619 22:26:49.082312 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:49.082330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:49.082445 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:49.082751 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:49.082754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:49.082965 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:49.083090 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:49.083098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.083898 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.083901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:49.084470 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:49.084478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.084881 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.084883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:49.084942 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:49.085424 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:49.085435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:49.087264 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
W20260619 22:26:49.087543 15037 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:49.082201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:44291 (local address 127.5.176.254:46509)
0619 22:26:49.082350 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:49.082354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:49.082387 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:26:49.082557 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:49.082563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:49.082628 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:49.082700 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:49.083201 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.083794 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.084571 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.084780 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.084853 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:49.087417 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:44291: BlockingRecv error: recv got EOF from 127.5.176.195:44291 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 22:26:49 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:49 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:49.710964 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:49.705548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:35761)
0619 22:26:49.705872 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:49.705889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:49.706036 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:49.706341 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:49.706344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:49.706551 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:49.706639 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:49.706644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.707409 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.707414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:49.707962 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:49.707970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.708443 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.708447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:49.708533 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:49.709021 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:49.709032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:49.710849 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
W20260619 22:26:49.711211 15041 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:49.705627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:35761 (local address 127.5.176.254:46509)
0619 22:26:49.705871 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:49.705877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:49.705972 (+ 95us) server_negotiation.cc:407] Connection header received
0619 22:26:49.706158 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:49.706162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:49.706216 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:49.706324 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:49.706753 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.707293 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.708079 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.708325 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.708437 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:49.711057 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:35761: BlockingRecv error: recv got EOF from 127.5.176.193:35761 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
Jun 19 22:26:49 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:49 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:49.892961 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:49.887261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:53789)
0619 22:26:49.887585 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:49.887598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:49.887736 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:49.888024 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:49.888027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:49.888230 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:49.888318 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:49.888324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.889100 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.889105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:49.889656 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:49.889664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.890167 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.890171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:49.890263 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:49.890741 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:49.890752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:49.892794 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
W20260619 22:26:49.893361 15041 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:49.887341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:53789 (local address 127.5.176.254:46509)
0619 22:26:49.887532 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:49.887537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:49.887674 (+ 137us) server_negotiation.cc:407] Connection header received
0619 22:26:49.887836 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:49.887848 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:49.887895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:49.887992 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:49.888448 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.888988 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.889775 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:49.890019 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:49.890130 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:49.892967 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:53789: BlockingRecv error: recv got EOF from 127.5.176.194:53789 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 22:26:50 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:50 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:50.094310 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:50.088263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:47849)
0619 22:26:50.088562 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:50.088577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:50.088704 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:50.088995 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:50.088998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:50.089177 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:50.089272 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:50.089277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.090026 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.090029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:50.090648 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:50.090656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.091203 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.091206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:50.091271 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:50.091946 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:50.091963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:50.094099 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
W20260619 22:26:50.094551 15041 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:50.088317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:47849 (local address 127.5.176.254:46509)
0619 22:26:50.088464 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:50.088475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:50.088647 (+ 172us) server_negotiation.cc:407] Connection header received
0619 22:26:50.088788 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:50.088790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:50.088837 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:50.088914 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:50.089384 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.089903 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.090793 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.091064 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.091185 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:50.094338 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:47849: BlockingRecv error: recv got EOF from 127.5.176.195:47849 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 22:26:50 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:50 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:50.717129 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:50.711751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:46613)
0619 22:26:50.712062 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:50.712075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:50.712185 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:50.712439 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:50.712442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:50.712617 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:50.712713 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:50.712718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.713439 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.713441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:50.714019 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:50.714028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.714476 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.714479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:50.714542 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:50.715078 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:50.715090 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:50.717009 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
W20260619 22:26:50.717279 15045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:50.711805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:46613 (local address 127.5.176.254:46509)
0619 22:26:50.712085 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:50.712088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:50.712127 (+ 39us) server_negotiation.cc:407] Connection header received
0619 22:26:50.712269 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:50.712271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:50.712322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:50.712398 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:50.712810 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.713337 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.714126 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.714378 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.714457 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:50.717172 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:46613: BlockingRecv error: recv got EOF from 127.5.176.193:46613 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
Jun 19 22:26:50 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:50 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:50.899956 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:50.893824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:60089)
0619 22:26:50.894105 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:50.894119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:50.894237 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:50.894545 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:50.894559 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:50.894809 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:50.894943 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:50.895006 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.896051 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.896053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:50.896595 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:50.896603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.896991 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.896993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:50.897053 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:50.897528 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:50.897539 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:50.899776 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
W20260619 22:26:50.900002 15045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:50.893925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:60089 (local address 127.5.176.254:46509)
0619 22:26:50.894063 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:50.894067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:50.894204 (+ 137us) server_negotiation.cc:407] Connection header received
0619 22:26:50.894338 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:50.894341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:50.894394 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:50.894488 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:50.895159 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.895953 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.896689 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:50.896895 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:50.896976 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:50.899903 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:60089: BlockingRecv error: recv got EOF from 127.5.176.194:60089 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 22:26:51 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:51 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:51.104192 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:51.095063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:41537)
0619 22:26:51.095343 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:51.095366 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:51.095461 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:51.095673 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:51.095675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:51.095847 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:51.095948 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:51.095953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.096728 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.096731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:51.097285 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:51.097292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.097681 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.097683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:51.097742 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:51.101809 (+ 4067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:51.101823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:51.104066 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
W20260619 22:26:51.104324 15045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:51.095111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:41537 (local address 127.5.176.254:46509)
0619 22:26:51.095248 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:51.095253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:51.095434 (+ 181us) server_negotiation.cc:407] Connection header received
0619 22:26:51.095526 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:51.095529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:51.095576 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:51.095642 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:51.096102 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.096636 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.097382 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.097589 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.097661 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:51.104208 (+ 6547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:41537: BlockingRecv error: recv got EOF from 127.5.176.195:41537 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 22:26:51 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:51 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:51.723907 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:51.718181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:45135)
0619 22:26:51.718478 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:51.718492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:51.718587 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:51.718852 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:51.718854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:51.719035 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:51.719124 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:51.719129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.719924 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.719926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:51.720475 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:51.720482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.720957 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.720959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:51.721019 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:51.721598 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:51.721610 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:51.723780 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
W20260619 22:26:51.724105 15049 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:51.718339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:45135 (local address 127.5.176.254:46509)
0619 22:26:51.718569 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:51.718574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:51.718588 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:26:51.718691 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:51.718693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:51.718736 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:51.718815 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:51.719243 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.719829 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.720594 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.720854 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.720943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:51.723928 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:45135: BlockingRecv error: recv got EOF from 127.5.176.193:45135 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
Jun 19 22:26:51 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:51 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:51.906014 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:51.900746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:57131)
0619 22:26:51.901032 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:51.901045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:51.901143 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:51.901379 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:51.901381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:51.901561 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:51.901658 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:51.901663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.902458 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.902461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:51.903024 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:51.903032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.903430 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.903432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:51.903494 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:51.904059 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:51.904072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:51.905886 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
W20260619 22:26:51.906189 15049 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:51.900889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:57131 (local address 127.5.176.254:46509)
0619 22:26:51.901053 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:51.901056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:51.901110 (+ 54us) server_negotiation.cc:407] Connection header received
0619 22:26:51.901226 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:51.901228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:51.901272 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:51.901342 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:51.901761 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.902349 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.903124 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:51.903329 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:51.903412 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:51.906016 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:57131: BlockingRecv error: recv got EOF from 127.5.176.194:57131 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 22:26:52 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:52 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:52.111529 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:52.105047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:55163)
0619 22:26:52.105371 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:52.105393 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:52.105546 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:52.105860 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:52.105862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:52.106076 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:52.106200 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:52.106206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.106979 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.106984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:52.107546 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:52.107554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.108026 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.108029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:52.108134 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:52.108889 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:52.108906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:52.111409 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":87,"threads_started":1}
W20260619 22:26:52.111799 15049 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:52.105118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:55163 (local address 127.5.176.254:46509)
0619 22:26:52.105335 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:52.105340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:52.105477 (+ 137us) server_negotiation.cc:407] Connection header received
0619 22:26:52.105656 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:52.105660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:52.105742 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:52.105835 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:52.106331 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.106864 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.107672 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.107905 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.108020 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:52.111668 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:55163: BlockingRecv error: recv got EOF from 127.5.176.195:55163 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 19 22:26:52 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:52 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:52.730257 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:52.724750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:42475)
0619 22:26:52.725091 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:52.725114 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:52.725210 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:52.725518 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:52.725521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:52.725726 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:52.725828 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:52.725834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.726669 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.726675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:52.727239 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:52.727246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.727699 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.727702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:52.727799 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:52.728292 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:52.728303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:52.730044 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
W20260619 22:26:52.730319 15053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:52.724864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:42475 (local address 127.5.176.254:46509)
0619 22:26:52.725189 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:52.725196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:52.725216 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:26:52.725341 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:52.725343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:52.725389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:52.725494 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:52.725949 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.726560 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.727369 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.727589 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.727695 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:52.730187 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:42475: BlockingRecv error: recv got EOF from 127.5.176.193:42475 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":62,"threads_started":1}
Jun 19 22:26:52 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:52 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:52.912007 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:52.906773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:35297)
0619 22:26:52.907073 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:52.907087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:52.907187 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:52.907404 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:52.907407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:52.907595 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:52.907688 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:52.907694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.908424 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.908426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:52.908985 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:52.908992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.909385 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.909387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:52.909466 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:52.909964 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:52.909975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:52.911881 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
W20260619 22:26:52.912197 15053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:52.906863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:35297 (local address 127.5.176.254:46509)
0619 22:26:52.907024 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:52.907029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:52.907203 (+ 174us) server_negotiation.cc:407] Connection header received
0619 22:26:52.907251 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:52.907253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:52.907299 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:52.907373 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:52.907799 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.908330 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.909083 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:52.909291 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:52.909364 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:52.912056 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:35297: BlockingRecv error: recv got EOF from 127.5.176.194:35297 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 22:26:53 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:53 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.195@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:53.118009 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:53.112418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.195:55009)
0619 22:26:53.112701 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:53.112715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:53.112865 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:53.113135 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:53.113138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:53.113380 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:53.113480 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:53.113486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.114267 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.114272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:53.114859 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:53.114867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.115354 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.115357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:53.115443 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:53.115931 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:53.115943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:53.117813 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
W20260619 22:26:53.118038 15053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:53.112498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:55009 (local address 127.5.176.254:46509)
0619 22:26:53.112647 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:53.112654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:53.112798 (+ 144us) server_negotiation.cc:407] Connection header received
0619 22:26:53.112964 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:53.112968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:53.113018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:53.113113 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:53.113596 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.114130 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.114989 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.115223 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.115349 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:53.117931 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.195:55009: BlockingRecv error: recv got EOF from 127.5.176.195:55009 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 22:26:53 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:53 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.193@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:53.736613 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:53.731332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.193:46001)
0619 22:26:53.731670 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:53.731685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:53.731776 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:53.732037 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:53.732039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:53.732216 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:53.732311 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:53.732315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.733025 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.733027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:53.733586 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:53.733592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.733985 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.733988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:53.734046 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:53.734658 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:53.734669 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:53.736450 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
W20260619 22:26:53.736708 15057 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:53.731501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:46001 (local address 127.5.176.254:46509)
0619 22:26:53.731732 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:53.731736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:53.731749 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:53.731878 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:53.731880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:53.731923 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:53.732003 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:53.732419 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.732937 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.733684 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.733896 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.733969 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:53.736567 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.193:46001: BlockingRecv error: recv got EOF from 127.5.176.193:46001 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":57,"threads_started":1}
Jun 19 22:26:53 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 22:26:53 dist-test-slave-bt27 krb5kdc[13999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.176.194@KRBTEST.COM for oryx/127.5.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 22:26:53.919041 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:53.913409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:46509 (local address 127.5.176.194:40369)
0619 22:26:53.913699 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:53.913731 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:53.913869 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:53.914185 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:53.914188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:53.914414 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:53.914511 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:53.914517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.915250 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.915252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:53.915872 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:53.915880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.916327 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.916329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:53.916397 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:53.916933 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:53.916944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:53.918915 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.176.254:46509: Server oryx/127.5.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
W20260619 22:26:53.919358 15057 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:53.913484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:40369 (local address 127.5.176.254:46509)
0619 22:26:53.913618 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:53.913624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:53.913889 (+ 265us) server_negotiation.cc:407] Connection header received
0619 22:26:53.913965 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:53.913969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:53.914028 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:53.914109 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:53.914626 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.915154 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.915993 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:53.916227 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:53.916315 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:53.919083 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.176.194:40369: BlockingRecv error: recv got EOF from 127.5.176.194:40369 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 22:26:54.005589 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14582
I20260619 22:26:54.011106 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14734
I20260619 22:26:54.016516 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14866
I20260619 22:26:54.021822 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 14512
2026-06-19T22:26:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18028 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T22:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:54Z Disabled control of system clock
I20260619 22:26:54.052388 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37097
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 22:26:54.163408 15062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:54.163673 15062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:54.163712 15062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:54.167652 15062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:54.167716 15062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:54.167732 15062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:54.167748 15062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:54.167763 15062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:54.172484 15062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:37097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:54.173523 15062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:54.174832 15062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:54.183432 15070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:54.183465 15068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:54.183449 15062 server_base.cc:1034] running on GCE node
W20260619 22:26:54.183486 15067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:54.184324 15062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:54.185493 15062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:54.186685 15062 hybrid_clock.cc:648] HybridClock initialized: now 1781908014186678 us; error 35 us; skew 500 ppm
I20260619 22:26:54.188711 15062 webserver.cc:460] Webserver started at http://127.5.176.254:43705/ using document root <none> and password file <none>
I20260619 22:26:54.189404 15062 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:54.189507 15062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:54.189779 15062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:54.192090 15062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "3e22afcf54be48a9895d06ccb537c7e7"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "da391ee1e2bb374dd03f2f4c319d7295"
server_key_iv: "6671b18d71e5b30ec8b37edae837f153"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.192719 15062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "3e22afcf54be48a9895d06ccb537c7e7"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "da391ee1e2bb374dd03f2f4c319d7295"
server_key_iv: "6671b18d71e5b30ec8b37edae837f153"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.196736 15062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:26:54.199465 15076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.200811 15062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:54.201074 15062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "3e22afcf54be48a9895d06ccb537c7e7"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "da391ee1e2bb374dd03f2f4c319d7295"
server_key_iv: "6671b18d71e5b30ec8b37edae837f153"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.201215 15062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:54.216629 15062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:54.217535 15062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:54.217772 15062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:54.225495 15062 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:37097
I20260619 22:26:54.225548 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:37097 every 8 connection(s)
I20260619 22:26:54.226527 15062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:54.228365 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15062
I20260619 22:26:54.228466 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:54.228776 5827 external_mini_cluster.cc:1428] Setting key f01334cbc8911d67fa1505661bb758bf
I20260619 22:26:54.230043 15129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:54.235988 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7: Bootstrap starting.
I20260619 22:26:54.238444 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.230302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.5.176.254:37097)
0619 22:26:54.231015 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:54.231030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:54.231068 (+ 38us) server_negotiation.cc:407] Connection header received
0619 22:26:54.231724 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:54.231740 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:54.232158 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:54.232541 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:54.232991 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.233965 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.234935 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.235431 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.235605 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.236558 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:54.236563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:54.236576 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:54.236989 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:54.237046 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:54.237066 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:54.237182 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:54.237317 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 22:26:54.237601 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":143,"threads_started":1}
I20260619 22:26:54.238510 15129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:54.239328 15129 log.cc:824] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:54.241587 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7: No bootstrap required, opened a new log
I20260619 22:26:54.245519 15129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e22afcf54be48a9895d06ccb537c7e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37097 } }
I20260619 22:26:54.245844 15129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:54.245918 15129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e22afcf54be48a9895d06ccb537c7e7, State: Initialized, Role: FOLLOWER
I20260619 22:26:54.246411 15129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e22afcf54be48a9895d06ccb537c7e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37097 } }
I20260619 22:26:54.246538 15129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:54.246628 15129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:54.246729 15129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:54.247723 15129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e22afcf54be48a9895d06ccb537c7e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37097 } }
I20260619 22:26:54.248092 15129 leader_election.cc:302] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e22afcf54be48a9895d06ccb537c7e7; no voters:
I20260619 22:26:54.248463 15129 leader_election.cc:288] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:54.248606 15134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:54.248873 15134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3e22afcf54be48a9895d06ccb537c7e7, State: Running, Role: LEADER
I20260619 22:26:54.249236 15134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e22afcf54be48a9895d06ccb537c7e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37097 } }
I20260619 22:26:54.249799 15129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:54.251063 15136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e22afcf54be48a9895d06ccb537c7e7. Latest consensus state: current_term: 1 leader_uuid: "3e22afcf54be48a9895d06ccb537c7e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e22afcf54be48a9895d06ccb537c7e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37097 } } }
I20260619 22:26:54.251209 15136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:54.251616 15143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:54.251938 15135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e22afcf54be48a9895d06ccb537c7e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e22afcf54be48a9895d06ccb537c7e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37097 } } }
I20260619 22:26:54.252112 15135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:54.254552 15143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:54.260722 15143 catalog_manager.cc:1347] Generated new cluster ID: 65d6a17c4f4b4f35993e91826764e3c0
I20260619 22:26:54.260820 15143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:54.272010 15143 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:54.273039 15143 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:54.285360 15143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e22afcf54be48a9895d06ccb537c7e7: Generated new TSK 0
I20260619 22:26:54.286183 15143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:54.297358 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37097
--builtin_ntp_servers=127.5.176.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 22:26:54.412487 15153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:54.412762 15153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:54.412844 15153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:54.417047 15153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:54.417117 15153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:54.417176 15153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:54.421887 15153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:54.423053 15153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:54.424623 15153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:54.434002 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:54.433988 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:54.434011 15161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:54.434234 15153 server_base.cc:1034] running on GCE node
I20260619 22:26:54.434711 15153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:54.435405 15153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:54.436615 15153 hybrid_clock.cc:648] HybridClock initialized: now 1781908014436601 us; error 35 us; skew 500 ppm
I20260619 22:26:54.438627 15153 webserver.cc:460] Webserver started at http://127.5.176.193:46791/ using document root <none> and password file <none>
I20260619 22:26:54.439219 15153 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:54.439312 15153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:54.439539 15153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:54.441576 15153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "62803cf971094453b4568ada15ad12ec"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "673722f18a07fa5bd1915a8885b95968"
server_key_iv: "48a8d7da851f83f83a8d199ac2a4b2e7"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.442203 15153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "62803cf971094453b4568ada15ad12ec"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "673722f18a07fa5bd1915a8885b95968"
server_key_iv: "48a8d7da851f83f83a8d199ac2a4b2e7"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.445971 15153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:26:54.448446 15167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.449723 15153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:54.449853 15153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "62803cf971094453b4568ada15ad12ec"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "673722f18a07fa5bd1915a8885b95968"
server_key_iv: "48a8d7da851f83f83a8d199ac2a4b2e7"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.449967 15153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:54.469708 15153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:54.470472 15153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:54.470677 15153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:54.471287 15153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:54.472223 15153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:54.472294 15153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.472364 15153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:54.472410 15153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.481491 15153 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:39005
I20260619 22:26:54.481508 15280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:39005 every 8 connection(s)
I20260619 22:26:54.482496 15153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:54.484122 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15153
I20260619 22:26:54.484264 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:54.484625 5827 external_mini_cluster.cc:1428] Setting key 4d1d08dba02dd071fbbb70a2af937342
I20260619 22:26:54.486611 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37097
--builtin_ntp_servers=127.5.176.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 22:26:54.493616 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.484493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:39987 (local address 127.5.176.254:37097)
0619 22:26:54.484653 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:54.484658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:54.485975 (+ 1317us) server_negotiation.cc:407] Connection header received
0619 22:26:54.487502 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:54.487509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:54.487579 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:54.487691 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:54.488933 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.489460 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.491368 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.491581 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.491665 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.493069 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:54.493073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:54.493075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:54.493089 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:54.493111 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:54.493119 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:54.493203 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:54.493420 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 22:26:54.493488 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:26:54.495025 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.485042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37097 (local address 127.5.176.193:39987)
0619 22:26:54.485817 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:54.485886 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:54.487283 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:54.487907 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:54.487917 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:54.488038 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:54.488771 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:54.488792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.489595 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.489602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:54.491239 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:54.491254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.491749 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.491755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:54.492012 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.492697 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:54.492700 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:54.492766 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:54.492876 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:54.493217 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:54.493229 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:54.493237 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:26:54.493393 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 22:26:54.493752 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":159,"threads_started":1}
I20260619 22:26:54.496562 15281 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37097
I20260619 22:26:54.496837 15281 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:54.497403 15281 heartbeater.cc:511] Master 127.5.176.254:37097 requested a full tablet report, sending...
I20260619 22:26:54.499071 15093 ts_manager.cc:194] Registered new tserver with Master: 62803cf971094453b4568ada15ad12ec (127.5.176.193:39005)
I20260619 22:26:54.500281 15093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.193:39987
W20260619 22:26:54.598413 15284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:54.598729 15284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:54.598800 15284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:54.602837 15284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:54.602931 15284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:54.603018 15284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:54.608034 15284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:54.609234 15284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:54.610692 15284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:54.620338 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:54.620330 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:54.620426 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:54.620657 15284 server_base.cc:1034] running on GCE node
I20260619 22:26:54.621074 15284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:54.621706 15284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:54.622886 15284 hybrid_clock.cc:648] HybridClock initialized: now 1781908014622868 us; error 26 us; skew 500 ppm
I20260619 22:26:54.624826 15284 webserver.cc:460] Webserver started at http://127.5.176.194:32937/ using document root <none> and password file <none>
I20260619 22:26:54.625423 15284 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:54.625486 15284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:54.625743 15284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:54.627861 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "0c628b68a5bd48afac9fddafa2e3674b"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "c3498499bd0523f1a1e9dcce9ae93092"
server_key_iv: "625b9f3080892f0156e4529a1815694d"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.628430 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "0c628b68a5bd48afac9fddafa2e3674b"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "c3498499bd0523f1a1e9dcce9ae93092"
server_key_iv: "625b9f3080892f0156e4529a1815694d"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.632226 15284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 22:26:54.635079 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.636420 15284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:54.636570 15284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "0c628b68a5bd48afac9fddafa2e3674b"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "c3498499bd0523f1a1e9dcce9ae93092"
server_key_iv: "625b9f3080892f0156e4529a1815694d"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.636708 15284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:54.652268 15284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:54.653119 15284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:54.653347 15284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:54.653995 15284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:54.655000 15284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:54.655073 15284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.655143 15284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:54.655193 15284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.663952 15284 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:42059
I20260619 22:26:54.663972 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:42059 every 8 connection(s)
I20260619 22:26:54.664894 15284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:54.673610 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15284
I20260619 22:26:54.673765 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:54.674127 5827 external_mini_cluster.cc:1428] Setting key e963aeb3972f09db8bc3f6e4b0c31ab8
I20260619 22:26:54.674360 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.666748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:50875 (local address 127.5.176.254:37097)
0619 22:26:54.666908 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:54.666912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:54.668062 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 22:26:54.669452 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:54.669455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:54.669507 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:54.669587 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:54.670520 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.671052 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.672199 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.672415 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.672501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.673785 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:54.673792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:54.673794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:54.673812 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:54.673836 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:54.673848 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:54.673946 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:54.674095 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 22:26:54.674198 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:26:54.675443 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.667202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37097 (local address 127.5.176.194:50875)
0619 22:26:54.667908 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:54.667952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:54.669219 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:54.669764 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:54.669771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:54.669870 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:54.670389 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:54.670401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.671177 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.671180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:54.672088 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:54.672097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.672591 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.672595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:54.672753 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.673458 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:54.673459 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:54.673503 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:54.673576 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:54.673946 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:54.673953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:54.673959 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:26:54.674068 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 22:26:54.674439 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":120,"threads_started":1}
I20260619 22:26:54.676383 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37097
--builtin_ntp_servers=127.5.176.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 22:26:54.676664 15412 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37097
I20260619 22:26:54.676934 15412 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:54.677480 15412 heartbeater.cc:511] Master 127.5.176.254:37097 requested a full tablet report, sending...
I20260619 22:26:54.678625 15093 ts_manager.cc:194] Registered new tserver with Master: 0c628b68a5bd48afac9fddafa2e3674b (127.5.176.194:42059)
I20260619 22:26:54.679210 15093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.194:50875
W20260619 22:26:54.788024 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:54.788306 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:54.788343 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:54.792656 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:54.792717 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:54.792809 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:54.797636 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:54.798785 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:54.800160 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:54.809966 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:54.810130 15415 server_base.cc:1034] running on GCE node
W20260619 22:26:54.809966 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:54.810387 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:54.810835 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:54.811497 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:54.812733 15415 hybrid_clock.cc:648] HybridClock initialized: now 1781908014812695 us; error 48 us; skew 500 ppm
I20260619 22:26:54.814751 15415 webserver.cc:460] Webserver started at http://127.5.176.195:35715/ using document root <none> and password file <none>
I20260619 22:26:54.815347 15415 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:54.815438 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:54.815702 15415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:54.817968 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "13a4fa4753434e439ca91f480051b522"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "8fa7c378841e2ed0ca7692eda7062da7"
server_key_iv: "9bd8068d726ba6b0f1139d993124b1a8"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.818607 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "13a4fa4753434e439ca91f480051b522"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "8fa7c378841e2ed0ca7692eda7062da7"
server_key_iv: "9bd8068d726ba6b0f1139d993124b1a8"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.822554 15415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:54.825335 15429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.826774 15415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:54.826920 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "13a4fa4753434e439ca91f480051b522"
format_stamp: "Formatted at 2026-06-19 22:26:54 on dist-test-slave-bt27"
server_key: "8fa7c378841e2ed0ca7692eda7062da7"
server_key_iv: "9bd8068d726ba6b0f1139d993124b1a8"
server_key_version: "encryptionkey@0"
I20260619 22:26:54.827049 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:54.848181 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:54.848953 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:54.849184 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:54.849825 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:54.850903 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:54.850970 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.851037 15415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:54.851087 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:54.860401 15415 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:40463
I20260619 22:26:54.860419 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:40463 every 8 connection(s)
I20260619 22:26:54.861357 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:54.863173 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15415
I20260619 22:26:54.863306 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:54.863646 5827 external_mini_cluster.cc:1428] Setting key a58de952ae3404fae05cb8c78d2c078d
I20260619 22:26:54.871341 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.863508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:44841 (local address 127.5.176.254:37097)
0619 22:26:54.863653 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:54.863657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:54.865067 (+ 1410us) server_negotiation.cc:407] Connection header received
0619 22:26:54.866634 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:54.866637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:54.866690 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:54.866764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:54.867650 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.868178 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.869306 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.869539 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.869651 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.870833 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:54.870839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:54.870846 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:54.870858 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:54.870877 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:54.870885 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:54.870964 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:54.871131 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 22:26:54.871197 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:26:54.872237 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:54.864017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37097 (local address 127.5.176.195:44841)
0619 22:26:54.864901 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:54.864960 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:54.866433 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:54.866941 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:54.866948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:54.867035 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:54.867511 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:54.867523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.868301 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.868306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:54.869178 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:54.869186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:54.869689 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:54.869693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:54.869866 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:54.870539 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:54.870541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:54.870585 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:54.870651 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:54.870977 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:54.870987 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:54.870993 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:26:54.871130 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 22:26:54.871427 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":153,"threads_started":1}
I20260619 22:26:54.873347 15543 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37097
I20260619 22:26:54.873613 15543 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:54.874222 15543 heartbeater.cc:511] Master 127.5.176.254:37097 requested a full tablet report, sending...
I20260619 22:26:54.875311 15093 ts_manager.cc:194] Registered new tserver with Master: 13a4fa4753434e439ca91f480051b522 (127.5.176.195:40463)
I20260619 22:26:54.875845 15093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.195:44841
I20260619 22:26:54.877539 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 22:26:54.880324 15132 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:54.879456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.5.176.254:37097)
0619 22:26:54.879598 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:54.879603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:54.879761 (+ 158us) server_negotiation.cc:407] Connection header received
0619 22:26:54.879860 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:54.879863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:54.879901 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:54.879958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:54.880178 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33296: BlockingRecv error: recv got EOF from 127.0.0.1:33296 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:26:54.880424 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15153
I20260619 22:26:54.886344 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15284
I20260619 22:26:54.891796 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15415
I20260619 22:26:54.897220 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15062
2026-06-19T22:26:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (874 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T22:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:54Z Disabled control of system clock
I20260619 22:26:54.926703 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38483
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 22:26:55.039335 15554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:55.039625 15554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:55.039662 15554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:55.043753 15554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:55.043808 15554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:55.043823 15554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:55.043841 15554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:55.043856 15554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:55.048712 15554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:38483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:55.049803 15554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:55.051213 15554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:55.059453 15560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:55.059500 15559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:55.059500 15562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:55.059836 15554 server_base.cc:1034] running on GCE node
I20260619 22:26:55.060387 15554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:55.061486 15554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:55.062671 15554 hybrid_clock.cc:648] HybridClock initialized: now 1781908015062642 us; error 42 us; skew 500 ppm
I20260619 22:26:55.064556 15554 webserver.cc:460] Webserver started at http://127.5.176.254:43963/ using document root <none> and password file <none>
I20260619 22:26:55.065132 15554 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:55.065192 15554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:55.065440 15554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:55.067530 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "2b99ef58baa14dc0ae7dadbd73d24841"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "f5c6e755cbb659f502df54894932f9f0"
server_key_iv: "b3a18d972259844b028ad74f7736a85c"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.068084 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "2b99ef58baa14dc0ae7dadbd73d24841"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "f5c6e755cbb659f502df54894932f9f0"
server_key_iv: "b3a18d972259844b028ad74f7736a85c"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.071753 15554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:26:55.074234 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.075533 15554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:55.075680 15554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "2b99ef58baa14dc0ae7dadbd73d24841"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "f5c6e755cbb659f502df54894932f9f0"
server_key_iv: "b3a18d972259844b028ad74f7736a85c"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.075819 15554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:55.080411 15554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:55.081023 15554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:55.081207 15554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:55.088160 15554 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:38483
I20260619 22:26:55.088203 15620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:38483 every 8 connection(s)
I20260619 22:26:55.089149 15554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:55.092377 15621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:55.092742 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15554
I20260619 22:26:55.092880 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:55.093174 5827 external_mini_cluster.cc:1428] Setting key dfeccd7fe19c73df28f57ea36318d3da
I20260619 22:26:55.098768 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.094558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.5.176.254:38483)
0619 22:26:55.095189 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:55.095210 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:55.095255 (+ 45us) server_negotiation.cc:407] Connection header received
0619 22:26:55.095950 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:55.095968 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:55.096331 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:55.096692 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:55.097435 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:55.097440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:55.097452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:55.097462 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:55.097512 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:55.097527 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:55.097680 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:55.097760 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 22:26:55.098003 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260619 22:26:55.099931 15621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841: Bootstrap starting.
I20260619 22:26:55.102435 15621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:55.103125 15621 log.cc:824] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:55.105499 15621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841: No bootstrap required, opened a new log
I20260619 22:26:55.109215 15621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38483 } }
I20260619 22:26:55.109539 15621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:55.109637 15621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b99ef58baa14dc0ae7dadbd73d24841, State: Initialized, Role: FOLLOWER
I20260619 22:26:55.110092 15621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38483 } }
I20260619 22:26:55.110276 15621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:55.110324 15621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:55.110400 15621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:55.111382 15621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38483 } }
I20260619 22:26:55.111748 15621 leader_election.cc:302] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b99ef58baa14dc0ae7dadbd73d24841; no voters:
I20260619 22:26:55.112177 15621 leader_election.cc:288] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:55.112314 15626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:55.112550 15626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [term 1 LEADER]: Becoming Leader. State: Replica: 2b99ef58baa14dc0ae7dadbd73d24841, State: Running, Role: LEADER
I20260619 22:26:55.112870 15626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38483 } }
I20260619 22:26:55.113543 15621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:55.114743 15628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b99ef58baa14dc0ae7dadbd73d24841. Latest consensus state: current_term: 1 leader_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38483 } } }
I20260619 22:26:55.114775 15627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b99ef58baa14dc0ae7dadbd73d24841" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38483 } } }
I20260619 22:26:55.114873 15628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:55.114877 15627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841 [sys.catalog]: This master's current role is: LEADER
W20260619 22:26:55.118773 15642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:26:55.118849 15642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:26:55.118934 15635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:55.119751 15635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:55.125572 15635 catalog_manager.cc:1347] Generated new cluster ID: e784839f4abe47919250820556e27eeb
I20260619 22:26:55.125686 15635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:55.145852 15635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:55.146270 15635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:55.151947 15635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b99ef58baa14dc0ae7dadbd73d24841: Generated new TSK 0
I20260619 22:26:55.152639 15635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:26:55.158090 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38483
--builtin_ntp_servers=127.5.176.212:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 22:26:55.269702 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:55.270025 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:55.270103 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:55.274312 15645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:55.274399 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:55.274516 15645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:55.279611 15645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:55.280761 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:55.282231 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:55.292033 15653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:55.292032 15650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:55.292133 15645 server_base.cc:1034] running on GCE node
W20260619 22:26:55.292032 15651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:55.292719 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:55.293403 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:55.294629 15645 hybrid_clock.cc:648] HybridClock initialized: now 1781908015294607 us; error 50 us; skew 500 ppm
I20260619 22:26:55.296654 15645 webserver.cc:460] Webserver started at http://127.5.176.193:34839/ using document root <none> and password file <none>
I20260619 22:26:55.297276 15645 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:55.297375 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:55.297619 15645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:55.299835 15645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "4b3b5ff80b01445d8151b56792235f40"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "6a0bbbb1455872783e4efb7bbcfef9cb"
server_key_iv: "123ce627ad22c34f313b38c4346da7a7"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.300412 15645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "4b3b5ff80b01445d8151b56792235f40"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "6a0bbbb1455872783e4efb7bbcfef9cb"
server_key_iv: "123ce627ad22c34f313b38c4346da7a7"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.304220 15645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:55.307030 15659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.308550 15645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:55.308691 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "4b3b5ff80b01445d8151b56792235f40"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "6a0bbbb1455872783e4efb7bbcfef9cb"
server_key_iv: "123ce627ad22c34f313b38c4346da7a7"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.308827 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:55.313632 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:55.314368 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:55.314559 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:55.315183 15645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:55.316115 15645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:55.316186 15645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.316267 15645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:55.316315 15645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.326766 15645 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:34391
I20260619 22:26:55.326787 15772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:34391 every 8 connection(s)
I20260619 22:26:55.327764 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:55.334282 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.329717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:38781 (local address 127.5.176.254:38483)
0619 22:26:55.330024 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:55.330028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:55.331055 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 22:26:55.332453 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:55.332457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:55.332500 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:55.332586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:55.333728 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:55.333732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:55.333736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:55.333755 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:55.333777 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:55.333786 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:55.333888 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:55.334039 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 22:26:55.334087 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:26:55.334560 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15645
I20260619 22:26:55.334702 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:55.334998 5827 external_mini_cluster.cc:1428] Setting key 4021919b6f7258521464d15196d4d3e1
I20260619 22:26:55.335366 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.330195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38483 (local address 127.5.176.193:38781)
0619 22:26:55.330901 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:55.330944 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:55.332212 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:55.332773 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:55.332781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:55.332877 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:55.333446 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:55.333448 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:55.333490 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:55.333560 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:55.333881 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:55.333890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:55.333898 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 22:26:55.334088 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 22:26:55.334387 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260619 22:26:55.336621 15773 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38483
I20260619 22:26:55.336884 15773 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:55.337088 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38483
--builtin_ntp_servers=127.5.176.212:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 22:26:55.337363 15773 heartbeater.cc:511] Master 127.5.176.254:38483 requested a full tablet report, sending...
I20260619 22:26:55.339105 15585 ts_manager.cc:194] Registered new tserver with Master: 4b3b5ff80b01445d8151b56792235f40 (127.5.176.193:34391)
W20260619 22:26:55.453255 15776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:55.453524 15776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:55.453565 15776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:55.457559 15776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:55.457612 15776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:55.457667 15776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:55.462410 15776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:55.463557 15776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:55.464956 15776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:55.474758 15782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:55.474758 15781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:55.474779 15784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:55.475085 15776 server_base.cc:1034] running on GCE node
I20260619 22:26:55.475492 15776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:55.476217 15776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:55.477406 15776 hybrid_clock.cc:648] HybridClock initialized: now 1781908015477389 us; error 25 us; skew 500 ppm
I20260619 22:26:55.479490 15776 webserver.cc:460] Webserver started at http://127.5.176.194:37341/ using document root <none> and password file <none>
I20260619 22:26:55.480154 15776 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:55.480227 15776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:55.480444 15776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:55.482820 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "c95bc6c61f5c4cbea0f6f825806225da"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "b2bd0bbb774d333df60951e36a2b59c6"
server_key_iv: "7d025fb92ae4833ae8c4386e3fc78ee7"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.483448 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "c95bc6c61f5c4cbea0f6f825806225da"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "b2bd0bbb774d333df60951e36a2b59c6"
server_key_iv: "7d025fb92ae4833ae8c4386e3fc78ee7"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.487424 15776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:26:55.490171 15790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.491513 15776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:55.491670 15776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "c95bc6c61f5c4cbea0f6f825806225da"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "b2bd0bbb774d333df60951e36a2b59c6"
server_key_iv: "7d025fb92ae4833ae8c4386e3fc78ee7"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.491803 15776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:55.496678 15776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:55.497332 15776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:55.497516 15776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:55.498078 15776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:55.499064 15776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:55.499110 15776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.499173 15776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:55.499217 15776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.508261 15776 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:38289
I20260619 22:26:55.508281 15903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:38289 every 8 connection(s)
I20260619 22:26:55.509208 15776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:55.514542 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15776
I20260619 22:26:55.514667 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:55.514987 5827 external_mini_cluster.cc:1428] Setting key 989721915d671917dc237bc9400173ec
I20260619 22:26:55.515949 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.511201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:37337 (local address 127.5.176.254:38483)
0619 22:26:55.511377 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:55.511382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:55.512505 (+ 1123us) server_negotiation.cc:407] Connection header received
0619 22:26:55.513945 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:55.513949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:55.514021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:55.514125 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:55.515437 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:55.515442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:55.515446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:55.515462 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:55.515485 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:55.515497 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:55.515611 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:55.515779 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 22:26:55.515822 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 22:26:55.516973 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.511625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38483 (local address 127.5.176.194:37337)
0619 22:26:55.512346 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:55.512386 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:55.513733 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:55.514352 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:55.514363 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:55.514491 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:55.515063 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:55.515066 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:55.515150 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:55.515237 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:55.515599 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:55.515608 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:55.515621 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 22:26:55.515807 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 22:26:55.516032 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":154,"threads_started":1}
I20260619 22:26:55.517099 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38483
--builtin_ntp_servers=127.5.176.212:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 22:26:55.518910 15904 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38483
I20260619 22:26:55.519186 15904 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:55.519706 15904 heartbeater.cc:511] Master 127.5.176.254:38483 requested a full tablet report, sending...
I20260619 22:26:55.520704 15585 ts_manager.cc:194] Registered new tserver with Master: c95bc6c61f5c4cbea0f6f825806225da (127.5.176.194:38289)
W20260619 22:26:55.631865 15907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:55.632181 15907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:55.632225 15907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:55.636550 15907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:55.636622 15907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:55.636684 15907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:55.644438 15907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:55.645537 15907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:55.646936 15907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:55.656399 15915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:55.656478 15912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:55.656584 15907 server_base.cc:1034] running on GCE node
W20260619 22:26:55.656478 15913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:55.657125 15907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:55.657735 15907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:55.658941 15907 hybrid_clock.cc:648] HybridClock initialized: now 1781908015658922 us; error 29 us; skew 500 ppm
I20260619 22:26:55.660941 15907 webserver.cc:460] Webserver started at http://127.5.176.195:33679/ using document root <none> and password file <none>
I20260619 22:26:55.661526 15907 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:55.661592 15907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:55.661808 15907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:55.663937 15907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "8dae02d689fa4fd4b32129bc303534ce"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "d76aee402e3cccf04cbdd109bbbd4b3a"
server_key_iv: "821ebed5f2f1cc35e225a015abb78269"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.664506 15907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "8dae02d689fa4fd4b32129bc303534ce"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "d76aee402e3cccf04cbdd109bbbd4b3a"
server_key_iv: "821ebed5f2f1cc35e225a015abb78269"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.668409 15907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:26:55.671227 15921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.672529 15907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:26:55.672626 15907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "8dae02d689fa4fd4b32129bc303534ce"
format_stamp: "Formatted at 2026-06-19 22:26:55 on dist-test-slave-bt27"
server_key: "d76aee402e3cccf04cbdd109bbbd4b3a"
server_key_iv: "821ebed5f2f1cc35e225a015abb78269"
server_key_version: "encryptionkey@0"
I20260619 22:26:55.672729 15907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:55.677366 15907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:55.677923 15907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:55.678056 15907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:55.678629 15907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:55.679461 15907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:55.679510 15907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.679581 15907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:55.679615 15907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:55.688439 15907 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:45263
I20260619 22:26:55.688455 16034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:45263 every 8 connection(s)
I20260619 22:26:55.689371 15907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:26:55.694712 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 15907
I20260619 22:26:55.694839 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:55.695134 5827 external_mini_cluster.cc:1428] Setting key fd40c46a0416e6da6697fb2391976110
I20260619 22:26:55.695595 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.691248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:54571 (local address 127.5.176.254:38483)
0619 22:26:55.691416 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:55.691420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:55.692436 (+ 1016us) server_negotiation.cc:407] Connection header received
0619 22:26:55.693685 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:55.693688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:55.693725 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:55.693804 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:55.695110 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:55.695114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:55.695117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:26:55.695134 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:55.695155 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:26:55.695163 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:55.695268 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:55.695411 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 22:26:55.695457 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 22:26:55.696645 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:55.691632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38483 (local address 127.5.176.195:54571)
0619 22:26:55.692284 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:55.692334 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:55.693482 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:55.693960 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:55.693967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:55.694051 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:55.694784 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:26:55.694786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:55.694827 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:26:55.694940 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:55.695247 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:55.695255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:55.695261 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:26:55.695457 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 22:26:55.695699 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260619 22:26:55.697825 16035 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38483
I20260619 22:26:55.698061 16035 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:55.698493 16035 heartbeater.cc:511] Master 127.5.176.254:38483 requested a full tablet report, sending...
I20260619 22:26:55.699446 15585 ts_manager.cc:194] Registered new tserver with Master: 8dae02d689fa4fd4b32129bc303534ce (127.5.176.195:45263)
I20260619 22:26:55.709244 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 22:26:55.712272 15624 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:26:55.711359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.5.176.254:38483)
0619 22:26:55.711504 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:55.711509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:55.711678 (+ 169us) server_negotiation.cc:407] Connection header received
0619 22:26:55.711760 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:55.711762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:55.711795 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:55.711859 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:55.712100 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38626: BlockingRecv error: recv got EOF from 127.0.0.1:38626 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:26:55.712390 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15645
I20260619 22:26:55.717758 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15776
I20260619 22:26:55.722790 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15907
I20260619 22:26:55.727916 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 15554
2026-06-19T22:26:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:55 dist-test-slave-bt27 krb5kdc[16045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:55 dist-test-slave-bt27 krb5kdc[16045](info): set up 2 sockets
Jun 19 22:26:55 dist-test-slave-bt27 krb5kdc[16045](info): commencing operation
krb5kdc: starting...
W20260619 22:26:56.779294 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:56 dist-test-slave-bt27 krb5kdc[16045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:26:56.902621 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38147
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:57.013031 16058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:57.013361 16058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:57.013434 16058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:57.017701 16058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:26:57.017829 16058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:57.017870 16058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:57.017925 16058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:26:57.017962 16058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:26:57.023502 16058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:38147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:57.024684 16058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:57.025728 16058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:57.034238 16064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.034238 16063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.034298 16066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:57.034471 16058 server_base.cc:1034] running on GCE node
I20260619 22:26:57.035069 16058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:57.036231 16058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:57.037420 16058 hybrid_clock.cc:648] HybridClock initialized: now 1781908017037405 us; error 26 us; skew 500 ppm
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:57.040426 16058 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:26:57.041375 16058 webserver.cc:460] Webserver started at http://127.5.176.254:46713/ using document root <none> and password file <none>
I20260619 22:26:57.041958 16058 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:57.042055 16058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:57.042341 16058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:57.044458 16058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "94afce0356d0431fa5ef9439b2361164"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "617ff21c263ee71067bb2ffb6cd39e68"
server_key_iv: "4ef4bf6162f4f1c581a51c23d2231ed2"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.045048 16058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "94afce0356d0431fa5ef9439b2361164"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "617ff21c263ee71067bb2ffb6cd39e68"
server_key_iv: "4ef4bf6162f4f1c581a51c23d2231ed2"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.048959 16058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:26:57.051568 16073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.052960 16058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:26:57.053120 16058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "94afce0356d0431fa5ef9439b2361164"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "617ff21c263ee71067bb2ffb6cd39e68"
server_key_iv: "4ef4bf6162f4f1c581a51c23d2231ed2"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.053248 16058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:57.081247 16058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:57.084468 16058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:57.084704 16058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:57.092043 16058 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:38147
I20260619 22:26:57.092342 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:38147 every 8 connection(s)
I20260619 22:26:57.093081 16058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:26:57.096043 16126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:57.099681 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 16058
I20260619 22:26:57.099800 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:26:57.100090 5827 external_mini_cluster.cc:1428] Setting key 4b55d8360c14cd3a4d9105d146f9b442
I20260619 22:26:57.102051 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164: Bootstrap starting.
I20260619 22:26:57.104817 16126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:57.105698 16126 log.cc:824] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:57.107870 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164: No bootstrap required, opened a new log
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:57.112356 16126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94afce0356d0431fa5ef9439b2361164" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38147 } }
I20260619 22:26:57.112592 16126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:57.112629 16126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94afce0356d0431fa5ef9439b2361164, State: Initialized, Role: FOLLOWER
I20260619 22:26:57.113163 16126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94afce0356d0431fa5ef9439b2361164" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38147 } }
I20260619 22:26:57.113355 16126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:26:57.113441 16126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:26:57.113548 16126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:57.115343 16126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94afce0356d0431fa5ef9439b2361164" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38147 } }
I20260619 22:26:57.115796 16126 leader_election.cc:302] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94afce0356d0431fa5ef9439b2361164; no voters:
I20260619 22:26:57.116153 16126 leader_election.cc:288] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:26:57.116285 16131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:57.116564 16131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [term 1 LEADER]: Becoming Leader. State: Replica: 94afce0356d0431fa5ef9439b2361164, State: Running, Role: LEADER
I20260619 22:26:57.116900 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.101624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.5.176.254:38147)
0619 22:26:57.102050 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.102071 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.102115 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:26:57.102945 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.102968 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.103404 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.103789 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:57.104493 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.106017 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.107056 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.107728 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.107887 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.110894 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:57.110900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:57.110914 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:57.110943 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:57.112925 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.113462 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.113468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.113475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.113664 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.113961 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.113964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.113966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.114525 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:57.115150 (+ 625us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.115497 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.115842 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260619 22:26:57.116940 16131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94afce0356d0431fa5ef9439b2361164" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38147 } }
I20260619 22:26:57.117508 16126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:26:57.118310 16131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94afce0356d0431fa5ef9439b2361164. Latest consensus state: current_term: 1 leader_uuid: "94afce0356d0431fa5ef9439b2361164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94afce0356d0431fa5ef9439b2361164" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38147 } } }
I20260619 22:26:57.118439 16131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:57.119347 16138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:26:57.120004 16132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94afce0356d0431fa5ef9439b2361164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94afce0356d0431fa5ef9439b2361164" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38147 } } }
I20260619 22:26:57.120137 16132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164 [sys.catalog]: This master's current role is: LEADER
I20260619 22:26:57.122597 16138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:26:57.128391 16138 catalog_manager.cc:1347] Generated new cluster ID: b1897543d6af4b7395900c35229829fd
I20260619 22:26:57.128476 16138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:26:57.149423 16138 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:26:57.150768 16138 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:26:57.159924 16138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164: Generated new TSK 0
I20260619 22:26:57.160828 16138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:26:57.190259 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38147
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:57.308295 16152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:57.308589 16152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:57.308634 16152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:57.312928 16152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:57.312983 16152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:57.313081 16152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:26:57.318778 16152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:57.319974 16152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:57.321008 16152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:57.330667 16157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.330672 16158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:57.330888 16152 server_base.cc:1034] running on GCE node
W20260619 22:26:57.330672 16160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:57.331354 16152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:57.331965 16152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:57.333153 16152 hybrid_clock.cc:648] HybridClock initialized: now 1781908017333131 us; error 33 us; skew 500 ppm
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:57.336217 16152 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:26:57.337152 16152 webserver.cc:460] Webserver started at http://127.5.176.193:39093/ using document root <none> and password file <none>
I20260619 22:26:57.337765 16152 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:57.337862 16152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:57.338095 16152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:57.340350 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "8a61b9466cd24c958aaa8230ca1d027b"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "b744fcd380a2d8747152ae5a09c26aca"
server_key_iv: "a5ad38dff67b0f3cc93f4f9fd63e648e"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.341029 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "8a61b9466cd24c958aaa8230ca1d027b"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "b744fcd380a2d8747152ae5a09c26aca"
server_key_iv: "a5ad38dff67b0f3cc93f4f9fd63e648e"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.345083 16152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 22:26:57.347731 16167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.349146 16152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:26:57.349304 16152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "8a61b9466cd24c958aaa8230ca1d027b"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "b744fcd380a2d8747152ae5a09c26aca"
server_key_iv: "a5ad38dff67b0f3cc93f4f9fd63e648e"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.349431 16152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:57.369268 16152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:57.372681 16152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:57.372896 16152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:57.373512 16152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:57.374361 16152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:57.374440 16152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.374516 16152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:57.374560 16152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.383450 16152 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:39809
I20260619 22:26:57.383472 16280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:39809 every 8 connection(s)
I20260619 22:26:57.384387 16152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:26:57.390434 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 16152
I20260619 22:26:57.390588 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:26:57.390926 5827 external_mini_cluster.cc:1428] Setting key 9d6ed6f9aa88f25e5b78847023e840e0
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:57.399852 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.386188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:49947 (local address 127.5.176.254:38147)
0619 22:26:57.386305 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.386310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.387127 (+ 817us) server_negotiation.cc:407] Connection header received
0619 22:26:57.388417 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.388420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.388466 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.388540 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:57.389674 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.390432 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.392173 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.392516 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.392647 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.395632 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:57.395640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:57.395645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:57.395677 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:57.397239 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.397956 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.397959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.397960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.398045 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.398534 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.398537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.398538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.398749 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:57.398836 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.399587 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.399721 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 22:26:57.401059 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.386514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38147 (local address 127.5.176.193:49947)
0619 22:26:57.386975 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:57.387017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:57.388217 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:57.388721 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:57.388727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:57.389067 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:57.389523 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.389535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.390591 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.390595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.392005 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.392015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.392685 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.392689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.392898 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.393605 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:57.393625 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:57.395453 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:57.397402 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:57.397411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:57.397431 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:57.397837 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:57.398171 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:57.398176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:57.398179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:57.398392 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:57.398870 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:57.398882 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:57.399277 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 22:26:57.399510 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 22:26:57.399917 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:26:57.402411 16281 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38147
I20260619 22:26:57.402658 16281 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:57.403199 16281 heartbeater.cc:511] Master 127.5.176.254:38147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:26:57.404834 16090 ts_manager.cc:194] Registered new tserver with Master: 8a61b9466cd24c958aaa8230ca1d027b (127.5.176.193:39809)
I20260619 22:26:57.406350 16090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:49947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:26:57.419267 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38147
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:57.530632 16286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:57.530884 16286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:57.530928 16286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:57.534988 16286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:57.535044 16286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:57.535130 16286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:26:57.540388 16286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:57.541405 16286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:57.542465 16286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:57.552131 16292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.552090 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.552073 16294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:57.552341 16286 server_base.cc:1034] running on GCE node
I20260619 22:26:57.552794 16286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:57.553556 16286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:57.554742 16286 hybrid_clock.cc:648] HybridClock initialized: now 1781908017554719 us; error 34 us; skew 500 ppm
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:57.557497 16286 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:26:57.558420 16286 webserver.cc:460] Webserver started at http://127.5.176.194:34211/ using document root <none> and password file <none>
I20260619 22:26:57.558972 16286 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:57.559058 16286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:57.559307 16286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:57.561301 16286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "2aec2ad51ba44cc889d9904b28349573"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "bd14596719a3746719cd9b773f431d18"
server_key_iv: "109a4c60860a12a1f4c13aa3f3ea42a0"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.561868 16286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "2aec2ad51ba44cc889d9904b28349573"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "bd14596719a3746719cd9b773f431d18"
server_key_iv: "109a4c60860a12a1f4c13aa3f3ea42a0"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.565639 16286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:26:57.568141 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.569489 16286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:26:57.569655 16286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "2aec2ad51ba44cc889d9904b28349573"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "bd14596719a3746719cd9b773f431d18"
server_key_iv: "109a4c60860a12a1f4c13aa3f3ea42a0"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.569775 16286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:57.601562 16286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:57.605453 16286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:57.605743 16286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:57.606551 16286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:57.607450 16286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:57.607502 16286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.607578 16286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:57.607626 16286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.617583 16286 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:43887
I20260619 22:26:57.617656 16414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:43887 every 8 connection(s)
I20260619 22:26:57.618858 16286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:26:57.626371 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 16286
I20260619 22:26:57.626469 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:26:57.626755 5827 external_mini_cluster.cc:1428] Setting key 973e734d33895e4d33e7b15d15693732
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:57.633400 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.620967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:42175 (local address 127.5.176.254:38147)
0619 22:26:57.621110 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.621113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.622006 (+ 893us) server_negotiation.cc:407] Connection header received
0619 22:26:57.623381 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.623383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.623430 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.623507 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:57.624690 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.625262 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.626568 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.626804 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.626938 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.629679 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:57.629683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:57.629685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:57.629710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:57.631296 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.631836 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.631839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.631841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.631923 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.632276 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.632279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.632281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.632470 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:57.632578 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.633102 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.633227 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:26:57.634325 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.621365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38147 (local address 127.5.176.194:42175)
0619 22:26:57.621842 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:57.621894 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:57.623180 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:57.623680 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:57.623687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:57.624050 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:57.624546 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.624560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.625404 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.625407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.626417 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.626452 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.627020 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.627025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.627241 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.627897 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:57.627914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:57.629495 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:57.631419 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:57.631426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:57.631444 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:57.631719 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:57.632021 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:57.632024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:57.632026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:57.632173 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:57.632577 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:57.632585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:57.632858 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 22:26:57.633061 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 22:26:57.633397 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260619 22:26:57.635576 16415 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38147
I20260619 22:26:57.635836 16415 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:26:57.636379 16415 heartbeater.cc:511] Master 127.5.176.254:38147 requested a full tablet report, sending...
I20260619 22:26:57.637475 16090 ts_manager.cc:194] Registered new tserver with Master: 2aec2ad51ba44cc889d9904b28349573 (127.5.176.194:43887)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:26:57.638475 16090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:42175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:26:57.657330 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38147
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:26:57.768375 16420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:26:57.768640 16420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:26:57.768684 16420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:26:57.772768 16420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:26:57.772823 16420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:26:57.772933 16420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:26:57.778239 16420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:26:57.779235 16420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:26:57.780189 16420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:26:57.789644 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.789717 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:26:57.789638 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:26:57.789966 16420 server_base.cc:1034] running on GCE node
I20260619 22:26:57.790403 16420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:26:57.791038 16420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:26:57.792266 16420 hybrid_clock.cc:648] HybridClock initialized: now 1781908017792249 us; error 28 us; skew 500 ppm
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:26:57.795107 16420 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:26:57.795982 16420 webserver.cc:460] Webserver started at http://127.5.176.195:37101/ using document root <none> and password file <none>
I20260619 22:26:57.796517 16420 fs_manager.cc:359] Metadata directory not provided
I20260619 22:26:57.796576 16420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:26:57.796742 16420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:26:57.798897 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "0f68f50f351743b392d04d056f81c15e"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "166f2c88619ae28c66b90571defaf913"
server_key_iv: "0f13fa4022793061baaca7abf25097f3"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.799424 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "0f68f50f351743b392d04d056f81c15e"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "166f2c88619ae28c66b90571defaf913"
server_key_iv: "0f13fa4022793061baaca7abf25097f3"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.803274 16420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:26:57.805842 16435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.807293 16420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:26:57.807437 16420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "0f68f50f351743b392d04d056f81c15e"
format_stamp: "Formatted at 2026-06-19 22:26:57 on dist-test-slave-bt27"
server_key: "166f2c88619ae28c66b90571defaf913"
server_key_iv: "0f13fa4022793061baaca7abf25097f3"
server_key_version: "encryptionkey@0"
I20260619 22:26:57.807564 16420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:26:57.828612 16420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:26:57.832091 16420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:26:57.832333 16420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:26:57.832955 16420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:26:57.833806 16420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:26:57.833853 16420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.833894 16420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:26:57.833953 16420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:26:57.843179 16420 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42885
I20260619 22:26:57.843247 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42885 every 8 connection(s)
I20260619 22:26:57.844095 16420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
Jun 19 22:26:57 dist-test-slave-bt27 krb5kdc[16045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:26:57.854297 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 16420
I20260619 22:26:57.854460 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:26:57.854825 5827 external_mini_cluster.cc:1428] Setting key 3c4506a24bb0c8a64c932f5bf4d0d339
I20260619 22:26:57.858886 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.846022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:38791 (local address 127.5.176.254:38147)
0619 22:26:57.846228 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.846234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.847079 (+ 845us) server_negotiation.cc:407] Connection header received
0619 22:26:57.848533 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.848536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.848588 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.848720 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:57.849948 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.850593 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.851774 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.851994 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.852088 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.855058 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:57.855065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:57.855067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:57.855096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:57.856637 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.857196 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.857200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.857202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.857317 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.857662 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.857667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.857668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.857915 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:57.858019 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.858593 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.858722 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:26:57.859704 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.846442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38147 (local address 127.5.176.195:38791)
0619 22:26:57.846926 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:57.846968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:57.848291 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:57.848882 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:57.848912 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:57.849312 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:26:57.849798 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.849811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.850740 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.850743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.851648 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.851657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.852175 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.852178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.852375 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.852937 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:26:57.852953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:26:57.854774 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:26:57.856789 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:57.856795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:57.856811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:57.857082 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:57.857409 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:26:57.857412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:26:57.857414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:26:57.857560 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:26:57.858017 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:26:57.858024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:26:57.858355 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 22:26:57.858541 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 22:26:57.858873 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
I20260619 22:26:57.860809 16549 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38147
I20260619 22:26:57.861102 16549 heartbeater.cc:464] Registering TS with master...
I20260619 22:26:57.861636 16549 heartbeater.cc:511] Master 127.5.176.254:38147 requested a full tablet report, sending...
I20260619 22:26:57.862706 16090 ts_manager.cc:194] Registered new tserver with Master: 0f68f50f351743b392d04d056f81c15e (127.5.176.195:42885)
I20260619 22:26:57.863266 16090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:38791
I20260619 22:26:57.868571 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:26:57.877518 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.870539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.5.176.254:38147)
0619 22:26:57.870678 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.870682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.870839 (+ 157us) server_negotiation.cc:407] Connection header received
0619 22:26:57.870934 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.870937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.870973 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.871045 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:26:57.871673 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.872187 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.873044 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.873327 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.873419 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.874847 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:26:57.874850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:26:57.874852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:26:57.874874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:26:57.876100 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.876469 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.876471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.876472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.876542 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:26:57.876761 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:26:57.876763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:26:57.876764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:26:57.876964 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:26:57.877065 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.877311 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.877401 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:26:57.880510 16090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:26:57.883558 16090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:26:57.898798 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.892274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.5.176.195:42885)
0619 22:26:57.893308 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.893322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.893350 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:26:57.893435 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.893441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.893672 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.894278 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:57.894450 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.895611 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.897126 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.897990 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.898096 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.898283 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.898521 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.898643 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":133,"threads_started":1}
I20260619 22:26:57.898829 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.892096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42885 (local address 127.0.0.1:47694)
0619 22:26:57.892673 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:57.892724 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:57.892870 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:57.893888 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:57.893894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:57.893920 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:57.894072 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.894082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.895749 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.895753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.896811 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.896820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.898125 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.898129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.898276 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.898319 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 22:26:57.898490 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 22:26:57.898633 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":2,"spinlock_wait_cycles":7040,"thread_start_us":74,"threads_started":1}
I20260619 22:26:57.899065 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.892286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39809 (local address 127.0.0.1:39346)
0619 22:26:57.892729 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:57.892743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:57.892870 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:57.894872 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:57.894875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:57.894895 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:57.895028 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.895033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.896252 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.896256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.897555 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.897566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.898657 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.898661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.898772 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.898799 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:26:57.898853 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 22:26:57.898927 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":58,"spinlock_wait_cycles":4864,"thread_start_us":134,"threads_started":1}
I20260619 22:26:57.899979 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.893614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43887 (local address 127.0.0.1:59926)
0619 22:26:57.894393 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:57.894412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:57.894570 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:57.895292 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:57.895295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:57.895313 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:57.895409 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.895415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.896579 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.896582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.897707 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:57.897718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.899583 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.899586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:57.899660 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.899679 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 22:26:57.899782 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 22:26:57.899854 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":141,"threads_started":1}
I20260619 22:26:57.900054 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.893203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.5.176.194:43887)
0619 22:26:57.893654 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.893695 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.894496 (+ 801us) server_negotiation.cc:407] Connection header received
0619 22:26:57.894729 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.894735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.894942 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.895075 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:57.895533 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.896410 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.897840 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.898686 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.898796 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.899523 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.899811 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.899901 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260619 22:26:57.899721 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:57.894019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.5.176.193:39809)
0619 22:26:57.894393 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:57.894408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:57.894432 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:26:57.894506 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:57.894511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:57.894708 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:57.895147 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:57.895264 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.896107 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.897716 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:57.898524 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:57.899318 (+ 794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:57.899451 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:57.899495 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 22:26:57.899574 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":168,"threads_started":1}
I20260619 22:26:57.902026 16483 tablet_service.cc:1467] Processing CreateTablet for tablet c387489f1d1844ea81ca0b4d5058f729 (DEFAULT_TABLE table=test-table [id=2abdee21a88f42a0b8620b62fc927546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:57.902529 16349 tablet_service.cc:1467] Processing CreateTablet for tablet c387489f1d1844ea81ca0b4d5058f729 (DEFAULT_TABLE table=test-table [id=2abdee21a88f42a0b8620b62fc927546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:57.903199 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c387489f1d1844ea81ca0b4d5058f729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:57.902922 16215 tablet_service.cc:1467] Processing CreateTablet for tablet c387489f1d1844ea81ca0b4d5058f729 (DEFAULT_TABLE table=test-table [id=2abdee21a88f42a0b8620b62fc927546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:26:57.903544 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c387489f1d1844ea81ca0b4d5058f729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:57.903846 16215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c387489f1d1844ea81ca0b4d5058f729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:26:57.910178 16566 tablet_bootstrap.cc:492] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: Bootstrap starting.
I20260619 22:26:57.910956 16567 tablet_bootstrap.cc:492] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b: Bootstrap starting.
I20260619 22:26:57.912961 16567 tablet_bootstrap.cc:654] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:57.913424 16566 tablet_bootstrap.cc:654] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:57.913939 16567 log.cc:824] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:57.914280 16566 log.cc:824] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:57.914522 16568 tablet_bootstrap.cc:492] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Bootstrap starting.
I20260619 22:26:57.916157 16567 tablet_bootstrap.cc:492] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b: No bootstrap required, opened a new log
I20260619 22:26:57.916381 16567 ts_tablet_manager.cc:1397] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:26:57.916960 16566 tablet_bootstrap.cc:492] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: No bootstrap required, opened a new log
I20260619 22:26:57.917174 16568 tablet_bootstrap.cc:654] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Neither blocks nor log segments found. Creating new log.
I20260619 22:26:57.917308 16566 ts_tablet_manager.cc:1397] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 22:26:57.918081 16568 log.cc:824] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:26:57.920238 16568 tablet_bootstrap.cc:492] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: No bootstrap required, opened a new log
I20260619 22:26:57.920433 16568 ts_tablet_manager.cc:1397] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:26:57.920545 16567 raft_consensus.cc:348] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:57.920814 16567 raft_consensus.cc:374] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:57.920871 16567 raft_consensus.cc:729] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a61b9466cd24c958aaa8230ca1d027b, State: Initialized, Role: FOLLOWER
I20260619 22:26:57.920998 16566 raft_consensus.cc:348] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:57.921262 16566 raft_consensus.cc:374] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:57.921329 16566 raft_consensus.cc:729] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aec2ad51ba44cc889d9904b28349573, State: Initialized, Role: FOLLOWER
I20260619 22:26:57.921406 16567 consensus_queue.cc:260] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:57.922367 16566 consensus_queue.cc:260] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:57.922120 16281 heartbeater.cc:503] Master 127.5.176.254:38147 was elected leader, sending a full tablet report...
I20260619 22:26:57.922546 16567 ts_tablet_manager.cc:1428] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:26:57.923254 16415 heartbeater.cc:503] Master 127.5.176.254:38147 was elected leader, sending a full tablet report...
I20260619 22:26:57.923301 16566 ts_tablet_manager.cc:1428] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:26:57.925280 16568 raft_consensus.cc:348] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:57.925576 16568 raft_consensus.cc:374] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:26:57.925661 16568 raft_consensus.cc:729] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f68f50f351743b392d04d056f81c15e, State: Initialized, Role: FOLLOWER
I20260619 22:26:57.926103 16568 consensus_queue.cc:260] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:57.926762 16549 heartbeater.cc:503] Master 127.5.176.254:38147 was elected leader, sending a full tablet report...
I20260619 22:26:57.927101 16568 ts_tablet_manager.cc:1428] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:26:58.007616 16573 raft_consensus.cc:482] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:26:58.007864 16573 raft_consensus.cc:504] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:58.009133 16573 leader_election.cc:288] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f68f50f351743b392d04d056f81c15e (127.5.176.195:42885), 8a61b9466cd24c958aaa8230ca1d027b (127.5.176.193:39809)
I20260619 22:26:58.013350 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.009364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42885 (local address 127.5.176.194:56835)
0619 22:26:58.009479 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:58.009497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:58.009605 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:58.009927 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:58.009929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:58.009948 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:58.010072 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:58.010077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.011114 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.011121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:58.012200 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:58.012233 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.012877 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.012880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:58.013023 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:58.013040 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:26:58.013127 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 22:26:58.013200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":37}
I20260619 22:26:58.013350 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.009496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:56835 (local address 127.5.176.195:42885)
0619 22:26:58.009649 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:58.009654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:58.009671 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:26:58.009732 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:58.009735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:58.009809 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:58.009908 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:58.010225 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.010997 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.012335 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.012747 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.012877 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:58.012933 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:58.013144 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 22:26:58.013212 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:26:58.013360 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.009699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:36681 (local address 127.5.176.193:39809)
0619 22:26:58.009827 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:58.009832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:58.009973 (+ 141us) server_negotiation.cc:407] Connection header received
0619 22:26:58.010118 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:58.010123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:58.010227 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:58.010312 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:26:58.010568 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.011083 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.012342 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.012749 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.012888 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:58.012942 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:58.013159 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 22:26:58.013231 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:26:58.013377 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.009462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39809 (local address 127.5.176.194:36681)
0619 22:26:58.009864 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:58.009885 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:26:58.010004 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:26:58.010323 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:26:58.010325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:26:58.010340 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:26:58.010457 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:58.010462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.011216 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.011219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:58.012216 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:26:58.012225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.012880 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.012884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:26:58.013035 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:58.013052 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:26:58.013141 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 22:26:58.013218 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":102,"thread_start_us":141,"threads_started":1}
I20260619 22:26:58.014117 16235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c387489f1d1844ea81ca0b4d5058f729" candidate_uuid: "2aec2ad51ba44cc889d9904b28349573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a61b9466cd24c958aaa8230ca1d027b" is_pre_election: true
I20260619 22:26:58.014117 16503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c387489f1d1844ea81ca0b4d5058f729" candidate_uuid: "2aec2ad51ba44cc889d9904b28349573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f68f50f351743b392d04d056f81c15e" is_pre_election: true
I20260619 22:26:58.014473 16235 raft_consensus.cc:2454] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aec2ad51ba44cc889d9904b28349573 in term 0.
I20260619 22:26:58.014473 16503 raft_consensus.cc:2454] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aec2ad51ba44cc889d9904b28349573 in term 0.
I20260619 22:26:58.014957 16304 leader_election.cc:302] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f68f50f351743b392d04d056f81c15e, 2aec2ad51ba44cc889d9904b28349573; no voters:
I20260619 22:26:58.015247 16573 raft_consensus.cc:2781] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:26:58.015342 16573 raft_consensus.cc:482] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:26:58.015375 16573 raft_consensus.cc:3037] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:58.016453 16573 raft_consensus.cc:504] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:58.016851 16573 leader_election.cc:288] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [CANDIDATE]: Term 1 election: Requested vote from peers 0f68f50f351743b392d04d056f81c15e (127.5.176.195:42885), 8a61b9466cd24c958aaa8230ca1d027b (127.5.176.193:39809)
I20260619 22:26:58.017282 16503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c387489f1d1844ea81ca0b4d5058f729" candidate_uuid: "2aec2ad51ba44cc889d9904b28349573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f68f50f351743b392d04d056f81c15e"
I20260619 22:26:58.017367 16235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c387489f1d1844ea81ca0b4d5058f729" candidate_uuid: "2aec2ad51ba44cc889d9904b28349573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a61b9466cd24c958aaa8230ca1d027b"
I20260619 22:26:58.017469 16503 raft_consensus.cc:3037] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:58.017534 16235 raft_consensus.cc:3037] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:26:58.018990 16235 raft_consensus.cc:2454] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aec2ad51ba44cc889d9904b28349573 in term 1.
I20260619 22:26:58.019088 16503 raft_consensus.cc:2454] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aec2ad51ba44cc889d9904b28349573 in term 1.
I20260619 22:26:58.019345 16303 leader_election.cc:302] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aec2ad51ba44cc889d9904b28349573, 8a61b9466cd24c958aaa8230ca1d027b; no voters:
I20260619 22:26:58.019589 16573 raft_consensus.cc:2781] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:26:58.019860 16573 raft_consensus.cc:686] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 1 LEADER]: Becoming Leader. State: Replica: 2aec2ad51ba44cc889d9904b28349573, State: Running, Role: LEADER
I20260619 22:26:58.020243 16573 consensus_queue.cc:237] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } }
I20260619 22:26:58.023953 16089 catalog_manager.cc:5495] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aec2ad51ba44cc889d9904b28349573 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "2aec2ad51ba44cc889d9904b28349573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aec2ad51ba44cc889d9904b28349573" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43887 } health_report { overall_health: HEALTHY } } }
I20260619 22:26:58.053421 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.049441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.5.176.194:43887)
0619 22:26:58.049592 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 22:26:58.049596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:26:58.049609 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:26:58.049683 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:26:58.049685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:26:58.049739 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:26:58.049852 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:26:58.050234 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.050831 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.051728 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:26:58.051945 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:26:58.052037 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:26:58.052472 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:26:58.053005 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:26:58.053107 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:26:58.053217 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 22:26:58.053276 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:26:58.059996 16235 raft_consensus.cc:1261] T c387489f1d1844ea81ca0b4d5058f729 P 8a61b9466cd24c958aaa8230ca1d027b [term 1 FOLLOWER]: Refusing update from remote peer 2aec2ad51ba44cc889d9904b28349573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:58.060062 16503 raft_consensus.cc:1261] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 1 FOLLOWER]: Refusing update from remote peer 2aec2ad51ba44cc889d9904b28349573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:26:58.060806 16576 consensus_queue.cc:1035] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a61b9466cd24c958aaa8230ca1d027b" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:58.060999 16573 consensus_queue.cc:1035] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f68f50f351743b392d04d056f81c15e" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:26:58.070631 16580 mvcc.cc:205] Tried to move back new op lower bound from 7298695241965056000 to 7298695241813966848. Current Snapshot: MvccSnapshot[applied={T|T < 7298695241965056000}]
I20260619 22:26:58.078718 16090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:58.078939 16090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:26:58.081638 16090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94afce0356d0431fa5ef9439b2361164: Sending DeleteTablet for 3 replicas of tablet c387489f1d1844ea81ca0b4d5058f729
I20260619 22:26:58.082515 16483 tablet_service.cc:1514] Processing DeleteTablet for tablet c387489f1d1844ea81ca0b4d5058f729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:58 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:47694
I20260619 22:26:58.082515 16349 tablet_service.cc:1514] Processing DeleteTablet for tablet c387489f1d1844ea81ca0b4d5058f729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:58 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:59926
I20260619 22:26:58.082530 16215 tablet_service.cc:1514] Processing DeleteTablet for tablet c387489f1d1844ea81ca0b4d5058f729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:26:58 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:39346
I20260619 22:26:58.083292 16590 tablet_replica.cc:330] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573: stopping tablet replica
I20260619 22:26:58.083630 16590 raft_consensus.cc:2229] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:26:58.083881 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 16152
I20260619 22:26:58.084111 16590 raft_consensus.cc:2258] T c387489f1d1844ea81ca0b4d5058f729 P 2aec2ad51ba44cc889d9904b28349573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:26:58.084439 16591 tablet_replica.cc:330] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: stopping tablet replica
I20260619 22:26:58.084774 16591 raft_consensus.cc:2229] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:26:58.085078 16591 raft_consensus.cc:2258] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 22:26:58.090023 16075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:39809 (error 108)
I20260619 22:26:58.091660 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.091222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:39809 (local address 127.0.0.1:39358)
0619 22:26:58.091478 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:58.091562 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 22:26:58.091791 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 16286
W20260619 22:26:58.091881 16075 catalog_manager.cc:4584] TS 8a61b9466cd24c958aaa8230ca1d027b (127.5.176.193:39809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c387489f1d1844ea81ca0b4d5058f729: Network error: Client connection negotiation failed: client connection to 127.5.176.193:39809: connect: Connection refused (error 111)
I20260619 22:26:58.094225 16591 ts_tablet_manager.cc:1905] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:26:58.097180 16591 ts_tablet_manager.cc:1918] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:26:58.097283 16591 log.cc:1196] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/c387489f1d1844ea81ca0b4d5058f729
I20260619 22:26:58.097627 16591 ts_tablet_manager.cc:1939] T c387489f1d1844ea81ca0b4d5058f729 P 0f68f50f351743b392d04d056f81c15e: Deleting consensus metadata
I20260619 22:26:58.097645 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 16420
I20260619 22:26:58.098212 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:26:58.097869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43887 (local address 127.0.0.1:59940)
0619 22:26:58.098044 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 22:26:58.098098 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:43887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 22:26:58.104444 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 16058
2026-06-19T22:26:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2378 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:26:58 dist-test-slave-bt27 krb5kdc[16595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:26:58 dist-test-slave-bt27 krb5kdc[16595](info): set up 2 sockets
Jun 19 22:26:58 dist-test-slave-bt27 krb5kdc[16595](info): commencing operation
krb5kdc: starting...
W20260619 22:26:59.143206 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:26:59 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:26:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/rangeradmin_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/rangeradmin_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/rangerlookup_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/rangerlookup_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
I20260619 22:26:59.317440 5827 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4QvkS_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/postgres -l logfile start
2026-06-19 22:27:01.989 UTC [16625] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 22:27:01.989 UTC [16625] LOG: listening on IPv4 address "127.5.176.212", port 33941
2026-06-19 22:27:01.993 UTC [16625] LOG: listening on Unix socket "/tmp/.s.PGSQL.33941"
2026-06-19 22:27:02.015 UTC [16628] LOG: database system was shut down at 2026-06-19 22:27:00 UTC
2026-06-19 22:27:02.020 UTC [16625] LOG: database system is ready to accept connections
I20260619 22:27:02.978999 5827 mini_postgres.cc:96] Postgres bound to 33941
2026-06-19 22:27:02.985 UTC [16636] FATAL: database "slave" does not exist
127.5.176.212:33941 - accepting connections
I20260619 22:27:02.985844 5827 mini_ranger.cc:161] Starting Ranger...
I20260619 22:27:03.001071 5827 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 22:27:03.513279 5827 mini_ranger.cc:87] Created ranger Postgres database
I20260619 22:27:03.513375 5827 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-admin
2026-06-19 22:27:03,793 [I] DB FLAVOR :POSTGRES
2026-06-19 22:27:03,793 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:27:03,793 [I] Checking connection..
2026-06-19 22:27:03,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 22:27:04,036 [I] Checking connection passed.
2026-06-19 22:27:04,037 [I] --------- Verifying version history table ---------
2026-06-19 22:27:04,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:27:04,278 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:27:04,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:27:04,542 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:27:04,542 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 22:27:04,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 22:27:04.774 UTC [16700] WARNING: there is no transaction in progress
2026-06-19 22:27:04,780 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 22:27:04,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:27:05,021 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 22:27:05,021 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:27:05,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 22:27:05,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 22:27:05,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bt27.c.gcp-upstream.internal','N') ;"
2026-06-19 22:27:05,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:27:05,995 [I] Table x_portal_user does not exist in database ranger
2026-06-19 22:27:05,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:27:06,224 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 22:27:06,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:27:06,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:27:06,751 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 22:27:06,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 22:27:07.712 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.729 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.745 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.761 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.780 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.800 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.861 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.867 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.882 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.892 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.903 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.917 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.927 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.934 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.941 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:07.949 UTC [16808] WARNING: there is no transaction in progress
2026-06-19 22:27:08,590 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 22:27:08,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 22:27:08,828 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 22:27:08,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:27:09,088 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 22:27:09,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:27:09,327 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 22:27:09,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:27:09,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:27:09,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bt27.c.gcp-upstream.internal';"
2026-06-19 22:27:10,062 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 22:27:10,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 22:27:10,305 [I] DB_PATCHES have already been applied
I20260619 22:27:10.311283 5827 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-admin:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4QvkS_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 22:27:10.311352 5827 mini_ranger.cc:193] Using host: 127.5.176.212
I20260619 22:27:10.314808 5827 mini_ranger.cc:227] Ranger admin URL: http://127.5.176.212:53061
Jun 19, 2026 10:27:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:27:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:27:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:27:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:27:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 22:27:11.512827 5827 mini_ranger.cc:160] Time spent starting Ranger: real 8.527s user 0.003s sys 0.007s
Jun 19, 2026 10:27:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:27:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 10:27:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:27:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:27:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 22:27:32.646100 5827 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/rangerkms_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/rangerkms_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.176.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.176.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 22:27:32 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908052, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 22:27:32.731498 5827 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 22:27:32.741317 5827 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 22:27:33.543951 5827 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 22:27:33.544052 5827 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms
I20260619 22:27:33.544157 5827 mini_ranger_kms.cc:227] Using postgres at 127.5.176.212:33941
2026-06-19 22:27:33,780 [I] DB FLAVOR :POSTGRES
2026-06-19 22:27:33,780 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:27:33,780 [I] Checking connection
2026-06-19 22:27:33,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 22:27:34,043 [I] connection success
2026-06-19 22:27:34,043 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 22:27:34,043 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 22:27:34,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 22:27:34,370 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 22:27:34,371 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:27:34,371 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 22:27:34,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:33941/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 22:27:34,671 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 22:27:35.150818 5827 mini_ranger_kms.cc:310] Created kms service
I20260619 22:27:35.352209 5827 mini_ranger_kms.cc:326] Created kudu user
I20260619 22:27:35.401532 5827 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 22:27:35 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908052, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:27:35.799919 5827 mini_ranger_kms.cc:384] Added ranger policy
I20260619 22:27:35.800117 5827 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms:/tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 22:27:35.800144 5827 mini_ranger_kms.cc:240] Using host: 127.5.176.212
I20260619 22:27:35.803911 5827 mini_ranger_kms.cc:276] Ranger KMS PID: 16988
I20260619 22:27:35.804066 5827 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.176.212:48419
Jun 19, 2026 10:27:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:27:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:27:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:27:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 22:27:38 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.176.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:27:38.087069 5827 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.356s user 0.008s sys 0.212s
I20260619 22:27:38.087203 5827 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 22:27:38.087327 5827 mini_ranger_kms.cc:401] 127.5.176.212:48419/kms/v1/keys
Jun 19 22:27:38 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.176.212@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
Jun 19 22:27:38 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.176.212@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
Jun 19 22:27:41 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:27:41.107142 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:40015
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:40015
--ranger_config_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:48419/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:27:41.225029 17039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:27:41.225360 17039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:27:41.225437 17039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:27:41.229909 17039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:27:41.230017 17039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:27:41.230057 17039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:27:41.230115 17039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:27:41.230173 17039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:27:41.235970 17039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.176.212:48419/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:40015
--ranger_config_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:27:41.237125 17039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:27:41.238255 17039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:27:41.248005 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:41.248090 17039 server_base.cc:1034] running on GCE node
W20260619 22:27:41.248026 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:27:41.248333 17047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:41.248905 17039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:27:41.250331 17039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:27:41.253626 17039 hybrid_clock.cc:648] HybridClock initialized: now 1781908061253603 us; error 43 us; skew 500 ppm
Jun 19 22:27:41 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908061, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:27:41.256659 17039 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:27:41.257683 17039 webserver.cc:460] Webserver started at http://127.5.176.254:35775/ using document root <none> and password file <none>
I20260619 22:27:41.258311 17039 fs_manager.cc:359] Metadata directory not provided
I20260619 22:27:41.258396 17039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:27:41.258610 17039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:27:41 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908061, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:27:41.309177 17039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "945fe8adf96f47bbb87eb2794e0bdcf9"
format_stamp: "Formatted at 2026-06-19 22:27:41 on dist-test-slave-bt27"
server_key: "1036d673063c28ce056d814918df2b67"
server_key_iv: "1540d1a17885990bdd1a9c97ccf400f1"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:41.309816 17039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "945fe8adf96f47bbb87eb2794e0bdcf9"
format_stamp: "Formatted at 2026-06-19 22:27:41 on dist-test-slave-bt27"
server_key: "1036d673063c28ce056d814918df2b67"
server_key_iv: "1540d1a17885990bdd1a9c97ccf400f1"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:41.313776 17039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 22:27:41.328594 17054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:41.330060 17039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:27:41.330293 17039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "945fe8adf96f47bbb87eb2794e0bdcf9"
format_stamp: "Formatted at 2026-06-19 22:27:41 on dist-test-slave-bt27"
server_key: "1036d673063c28ce056d814918df2b67"
server_key_iv: "1540d1a17885990bdd1a9c97ccf400f1"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:41.330447 17039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:27:41.347896 17039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:27:41.351027 17039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:27:41.351238 17039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:27:41.359653 17039 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:40015
I20260619 22:27:41.359691 17106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:40015 every 8 connection(s)
I20260619 22:27:41.360661 17039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:27:41.364225 17107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:27:41.365720 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 17039
I20260619 22:27:41.365872 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:27:41.366250 5827 external_mini_cluster.cc:1428] Setting key 3a1cfc592c1602e42f47ab6332f5014d
I20260619 22:27:41.371523 17107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9: Bootstrap starting.
I20260619 22:27:41.374337 17107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9: Neither blocks nor log segments found. Creating new log.
I20260619 22:27:41.375236 17107 log.cc:824] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:27:41 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:27:41.378126 17107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9: No bootstrap required, opened a new log
I20260619 22:27:41.381538 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:41.367834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.5.176.254:40015)
0619 22:27:41.368268 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:41.368285 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:41.368332 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:27:41.368986 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:41.369003 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:41.369333 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:41.369681 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:27:41.370553 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:41.371571 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:41.372544 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:41.373025 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:41.373196 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:41.376190 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:27:41.376197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:27:41.376217 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:27:41.376249 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:27:41.378715 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:41.379197 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:41.379203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:41.379208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:41.379325 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:41.379623 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:41.379628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:41.379630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:41.380030 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:27:41.380251 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:41.380490 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 22:27:41.380785 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":159,"threads_started":1}
I20260619 22:27:41.382058 17107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40015 } }
I20260619 22:27:41.382365 17107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:27:41.382449 17107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945fe8adf96f47bbb87eb2794e0bdcf9, State: Initialized, Role: FOLLOWER
I20260619 22:27:41.382913 17107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40015 } }
I20260619 22:27:41.383059 17107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:27:41.383139 17107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:27:41.383234 17107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:27:41.384351 17107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40015 } }
I20260619 22:27:41.384727 17107 leader_election.cc:302] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 945fe8adf96f47bbb87eb2794e0bdcf9; no voters:
I20260619 22:27:41.385089 17107 leader_election.cc:288] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:27:41.385200 17112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:27:41.385452 17112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [term 1 LEADER]: Becoming Leader. State: Replica: 945fe8adf96f47bbb87eb2794e0bdcf9, State: Running, Role: LEADER
I20260619 22:27:41.385812 17112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40015 } }
I20260619 22:27:41.386370 17107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:27:41.387172 17113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 945fe8adf96f47bbb87eb2794e0bdcf9. Latest consensus state: current_term: 1 leader_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40015 } } }
I20260619 22:27:41.387293 17113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [sys.catalog]: This master's current role is: LEADER
I20260619 22:27:41.387656 17114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945fe8adf96f47bbb87eb2794e0bdcf9" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40015 } } }
I20260619 22:27:41.387734 17114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9 [sys.catalog]: This master's current role is: LEADER
I20260619 22:27:41.388516 17107 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 22:27:41.393417 17121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:27:41.397398 17121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:27:41.407837 17121 catalog_manager.cc:1347] Generated new cluster ID: 1abcd2ffaf274f03926b45e056098242
I20260619 22:27:41.407917 17121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:27:41.430236 17121 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:27:41.431663 17121 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:27:41.439556 17121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9: Generated new TSK 0
I20260619 22:27:41.440523 17121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:27:42.197504 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:48419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:40015
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:27:42.330264 17152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:27:42.330742 17152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:27:42.330837 17152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:27:42.336553 17152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:27:42.336650 17152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:27:42.336752 17152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:27:42.342504 17152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.176.212:48419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:40015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:27:42.343896 17152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:27:42.345204 17152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:27:42.357415 17159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:27:42.357421 17162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:27:42.357420 17160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:42.358386 17152 server_base.cc:1034] running on GCE node
I20260619 22:27:42.358824 17152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:27:42.359645 17152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:27:42.360855 17152 hybrid_clock.cc:648] HybridClock initialized: now 1781908062360837 us; error 47 us; skew 500 ppm
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:27:42.364483 17152 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:27:42.365749 17152 webserver.cc:460] Webserver started at http://127.5.176.193:41833/ using document root <none> and password file <none>
I20260619 22:27:42.366573 17152 fs_manager.cc:359] Metadata directory not provided
I20260619 22:27:42.366686 17152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:27:42.367007 17152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:27:42.386236 17152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "19be37eca15440d5a9dd3d283e5c69d9"
format_stamp: "Formatted at 2026-06-19 22:27:42 on dist-test-slave-bt27"
server_key: "e321209d169e43112bdfe9e633789a8a"
server_key_iv: "74253cc87326649a6324f053ef5899de"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:42.386996 17152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "19be37eca15440d5a9dd3d283e5c69d9"
format_stamp: "Formatted at 2026-06-19 22:27:42 on dist-test-slave-bt27"
server_key: "e321209d169e43112bdfe9e633789a8a"
server_key_iv: "74253cc87326649a6324f053ef5899de"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:42.392305 17152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 22:27:42.414463 17169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:42.416301 17152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:27:42.416453 17152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "19be37eca15440d5a9dd3d283e5c69d9"
format_stamp: "Formatted at 2026-06-19 22:27:42 on dist-test-slave-bt27"
server_key: "e321209d169e43112bdfe9e633789a8a"
server_key_iv: "74253cc87326649a6324f053ef5899de"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:42.416618 17152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:27:42.443900 17152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:27:42.447826 17152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:27:42.448086 17152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:27:42.448865 17152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:27:42.449958 17152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:27:42.450059 17152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:42.450207 17152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:27:42.450289 17152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:42.463666 17152 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:38451
I20260619 22:27:42.464711 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:38451 every 8 connection(s)
I20260619 22:27:42.465042 17152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:27:42.470929 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 17152
I20260619 22:27:42.471061 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:27:42.471402 5827 external_mini_cluster.cc:1428] Setting key c90b0ab73cb4693b01f5c3cc1952b0a0
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:27:42.489745 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:42.469872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40015 (local address 127.5.176.193:33119)
0619 22:27:42.472068 (+ 2196us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:42.472124 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:42.475162 (+ 3038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:42.476027 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:42.476038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:42.476428 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:27:42.477136 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:42.477157 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.478399 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.478403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:42.479842 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:42.479852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.480424 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.480429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:42.480662 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:42.481519 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:27:42.481546 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:27:42.483493 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:27:42.485968 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:27:42.485978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:27:42.485998 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:27:42.486533 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:27:42.486915 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:27:42.486921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:27:42.486924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:27:42.487117 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:27:42.487680 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:27:42.487687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:27:42.488061 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 22:27:42.488279 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 22:27:42.488643 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"thread_start_us":152,"threads_started":1}
I20260619 22:27:42.490551 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:42.469359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:33119 (local address 127.5.176.254:40015)
0619 22:27:42.469639 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:42.469645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:42.474858 (+ 5213us) server_negotiation.cc:407] Connection header received
0619 22:27:42.475539 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:42.475543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:42.475607 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:42.475702 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:27:42.477331 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.478242 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.479994 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.480242 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.480378 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:42.483726 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:27:42.483732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:27:42.483739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:27:42.483788 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:27:42.485832 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:42.486661 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:42.486666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:42.486670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:42.486810 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:42.487243 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:42.487246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:42.487249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:42.487549 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:27:42.487681 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:42.490225 (+ 2544us) server_negotiation.cc:299] Negotiation successful
0619 22:27:42.490376 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260619 22:27:42.492378 17283 heartbeater.cc:347] Connected to a master server at 127.5.176.254:40015
I20260619 22:27:42.493028 17283 heartbeater.cc:464] Registering TS with master...
I20260619 22:27:42.493613 17283 heartbeater.cc:511] Master 127.5.176.254:40015 requested a full tablet report, sending...
I20260619 22:27:42.495486 17071 ts_manager.cc:194] Registered new tserver with Master: 19be37eca15440d5a9dd3d283e5c69d9 (127.5.176.193:38451)
I20260619 22:27:42.496999 17071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:33119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:27:42.518507 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:48419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:40015
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 22:27:42.675968 17291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:27:42.676440 17291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:27:42.676561 17291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:27:42.682822 17291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:27:42.682986 17291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:27:42.683112 17291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:27:42.689939 17291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.176.212:48419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:40015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:27:42.691200 17291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:27:42.692523 17291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:27:42.705688 17297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:27:42.705607 17300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:42.706362 17291 server_base.cc:1034] running on GCE node
W20260619 22:27:42.706615 17298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:42.707173 17291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:27:42.708043 17291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:27:42.710240 17291 hybrid_clock.cc:648] HybridClock initialized: now 1781908062710210 us; error 44 us; skew 500 ppm
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:27:42.714578 17291 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:27:42.715790 17291 webserver.cc:460] Webserver started at http://127.5.176.194:34677/ using document root <none> and password file <none>
I20260619 22:27:42.716535 17291 fs_manager.cc:359] Metadata directory not provided
I20260619 22:27:42.716622 17291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:27:42.716851 17291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:27:42.736058 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "1697a794fdf9428f8829fb9dd323cb55"
format_stamp: "Formatted at 2026-06-19 22:27:42 on dist-test-slave-bt27"
server_key: "6ff514bf4cf99e8345de867c676e0efe"
server_key_iv: "1bb30b625f9029cad3fd7c3e75b6802d"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:42.737022 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "1697a794fdf9428f8829fb9dd323cb55"
format_stamp: "Formatted at 2026-06-19 22:27:42 on dist-test-slave-bt27"
server_key: "6ff514bf4cf99e8345de867c676e0efe"
server_key_iv: "1bb30b625f9029cad3fd7c3e75b6802d"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:42.742477 17291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:27:42.765959 17307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:42.767694 17291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:27:42.767836 17291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "1697a794fdf9428f8829fb9dd323cb55"
format_stamp: "Formatted at 2026-06-19 22:27:42 on dist-test-slave-bt27"
server_key: "6ff514bf4cf99e8345de867c676e0efe"
server_key_iv: "1bb30b625f9029cad3fd7c3e75b6802d"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:42.768007 17291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:27:42.793469 17291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:27:42.797951 17291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:27:42.798228 17291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:27:42.799010 17291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:27:42.800141 17291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:27:42.800202 17291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:42.800271 17291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:27:42.800298 17291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:42.812696 17291 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:39919
I20260619 22:27:42.812778 17420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:39919 every 8 connection(s)
I20260619 22:27:42.813830 17291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:27:42.819783 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 17291
I20260619 22:27:42.819954 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:27:42.820256 5827 external_mini_cluster.cc:1428] Setting key 45df3e9566d3b4a96ff4ac564d4424d4
Jun 19 22:27:42 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:27:42.829486 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:42.815944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:34601 (local address 127.5.176.254:40015)
0619 22:27:42.816106 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:42.816111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:42.817096 (+ 985us) server_negotiation.cc:407] Connection header received
0619 22:27:42.818201 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:42.818205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:42.818266 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:42.818370 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:27:42.819551 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.820343 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.821943 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.822303 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.822415 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:42.825639 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:27:42.825643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:27:42.825646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:27:42.825672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:27:42.827185 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:42.827804 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:42.827808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:42.827810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:42.827892 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:42.828312 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:42.828317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:42.828319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:42.828517 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:27:42.828621 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:42.829178 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 22:27:42.829315 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:27:42.830806 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:42.816405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40015 (local address 127.5.176.194:34601)
0619 22:27:42.816897 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:42.816961 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:42.817824 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:42.818555 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:42.818562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:42.818843 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:27:42.819363 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:42.819377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.820519 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.820523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:42.821821 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:42.821831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:42.822492 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:42.822499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:42.822739 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:42.823504 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:27:42.823531 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:27:42.825424 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:27:42.827330 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:27:42.827337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:27:42.827354 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:27:42.827693 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:27:42.827994 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:27:42.827997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:27:42.827999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:27:42.828214 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:27:42.828635 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:27:42.828644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:27:42.828934 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 22:27:42.829143 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 22:27:42.829499 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:27:42.832055 17421 heartbeater.cc:347] Connected to a master server at 127.5.176.254:40015
I20260619 22:27:42.832295 17421 heartbeater.cc:464] Registering TS with master...
I20260619 22:27:42.832818 17421 heartbeater.cc:511] Master 127.5.176.254:40015 requested a full tablet report, sending...
I20260619 22:27:42.833959 17071 ts_manager.cc:194] Registered new tserver with Master: 1697a794fdf9428f8829fb9dd323cb55 (127.5.176.194:39919)
I20260619 22:27:42.834594 17071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:34601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:27:42.861364 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:48419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:40015
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:27:43 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
W20260619 22:27:43.029491 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:27:43.029927 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:27:43.030043 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:27:43.036257 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:27:43.036342 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:27:43.036484 17426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:27:43.044715 17426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.176.212:48419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:40015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:27:43.046267 17426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:27:43.047674 17426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:27:43.062431 17434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:27:43.062912 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:43.064023 17426 server_base.cc:1034] running on GCE node
W20260619 22:27:43.064771 17431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:27:43.065325 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:27:43.066207 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:27:43.067523 17426 hybrid_clock.cc:648] HybridClock initialized: now 1781908063067447 us; error 92 us; skew 500 ppm
Jun 19 22:27:43 dist-test-slave-bt27 krb5kdc[16595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:27:43.071166 17426 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:27:43.072257 17426 webserver.cc:460] Webserver started at http://127.5.176.195:42863/ using document root <none> and password file <none>
I20260619 22:27:43.072983 17426 fs_manager.cc:359] Metadata directory not provided
I20260619 22:27:43.073073 17426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:27:43.073318 17426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:27:43 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:27:43.095613 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58"
format_stamp: "Formatted at 2026-06-19 22:27:43 on dist-test-slave-bt27"
server_key: "50c0a6d5d0b44d6609334c176d14f417"
server_key_iv: "20f7f9cc1d50f0646c52ae3308d5bd29"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:43.096374 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58"
format_stamp: "Formatted at 2026-06-19 22:27:43 on dist-test-slave-bt27"
server_key: "50c0a6d5d0b44d6609334c176d14f417"
server_key_iv: "20f7f9cc1d50f0646c52ae3308d5bd29"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:43.103577 17426 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 22:27:43.142653 17441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:43.145133 17426 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260619 22:27:43.145311 17426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58"
format_stamp: "Formatted at 2026-06-19 22:27:43 on dist-test-slave-bt27"
server_key: "50c0a6d5d0b44d6609334c176d14f417"
server_key_iv: "20f7f9cc1d50f0646c52ae3308d5bd29"
server_key_version: "kuduclusterkey@0"
I20260619 22:27:43.145498 17426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:27:43.180799 17426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:27:43.197753 17426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:27:43.197985 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:27:43.198778 17426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:27:43.199918 17426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:27:43.199985 17426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:43.200052 17426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:27:43.200083 17426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:27:43.218681 17426 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42555
I20260619 22:27:43.219972 17426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:27:43.222254 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 17426
I20260619 22:27:43.222393 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:27:43.222749 5827 external_mini_cluster.cc:1428] Setting key 7aea8cfffa9e674c2319663d473ede3d
I20260619 22:27:43.224761 17554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42555 every 8 connection(s)
Jun 19 22:27:43 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:27:43.246472 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.224485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:35907 (local address 127.5.176.254:40015)
0619 22:27:43.224639 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.224647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.230707 (+ 6060us) server_negotiation.cc:407] Connection header received
0619 22:27:43.231923 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.231927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.231990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.232083 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:27:43.233785 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.234636 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.238023 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.238408 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.238482 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.241971 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:27:43.241975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:27:43.241979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:27:43.242009 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:27:43.243879 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:43.244531 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:43.244535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:43.244537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:43.244624 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:43.245057 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:43.245060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:43.245061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:43.245288 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:27:43.245380 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.246095 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.246269 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 22:27:43.249802 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.223437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40015 (local address 127.5.176.195:35907)
0619 22:27:43.230505 (+ 7068us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.230576 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.231663 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.232431 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:43.232441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:43.232832 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:27:43.233568 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.233591 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.234843 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.234846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.237884 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.237896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.238688 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.238691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.238890 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.239621 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:27:43.239644 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:27:43.241769 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:27:43.244008 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:27:43.244015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:27:43.244033 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:27:43.244393 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:27:43.244733 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:27:43.244739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:27:43.244744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:27:43.244941 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:27:43.245460 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:27:43.245467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:27:43.245799 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 22:27:43.246731 (+ 932us) client_negotiation.cc:240] Negotiation successful
0619 22:27:43.248681 (+ 1950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6797,"thread_start_us":133,"threads_started":1}
I20260619 22:27:43.251142 17555 heartbeater.cc:347] Connected to a master server at 127.5.176.254:40015
I20260619 22:27:43.251411 17555 heartbeater.cc:464] Registering TS with master...
I20260619 22:27:43.252053 17555 heartbeater.cc:511] Master 127.5.176.254:40015 requested a full tablet report, sending...
I20260619 22:27:43.253173 17071 ts_manager.cc:194] Registered new tserver with Master: ba8bc7aca4a649b0b63fb8d7ec65fa58 (127.5.176.195:42555)
I20260619 22:27:43.253906 17071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:35907
I20260619 22:27:43.260197 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:27:43.274582 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.264366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.5.176.254:40015)
0619 22:27:43.264482 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.264487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.265143 (+ 656us) server_negotiation.cc:407] Connection header received
0619 22:27:43.265362 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.265365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.265419 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.265478 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:27:43.266534 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.267365 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.268580 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.268903 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.268982 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.270902 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:27:43.270906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:27:43.270909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:27:43.270936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:27:43.272715 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:43.273196 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:43.273200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:43.273202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:43.273282 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:27:43.273595 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:27:43.273601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:27:43.273602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:27:43.273813 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:27:43.273922 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.274295 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.274422 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:27:43.278474 17071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:27:43.282989 17071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:27:43.325552 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.318162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42555 (local address 127.0.0.1:39586)
0619 22:27:43.318632 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.318690 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.318832 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.320058 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:43.320064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:43.320087 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:27:43.320229 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.320235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.322372 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.322375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.323822 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.323832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.325048 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.325052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.325162 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.325205 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 22:27:43.325286 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 22:27:43.325393 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":138,"threads_started":1}
I20260619 22:27:43.328513 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.319185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.5.176.195:42555)
0619 22:27:43.319567 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.319577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.319600 (+ 23us) server_negotiation.cc:407] Connection header received
0619 22:27:43.319674 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.319679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.319888 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.320377 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:27:43.320501 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.322242 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.324065 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.324914 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.328055 (+ 3141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.328215 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.328264 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.328361 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":107,"threads_started":1}
I20260619 22:27:43.331753 17487 tablet_service.cc:1467] Processing CreateTablet for tablet c2d271f2460947d1aa03e0262f6faa14 (DEFAULT_TABLE table=test-table [id=800e13d498e44ac7b11b9dfe498d7b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:27:43.332970 17487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d271f2460947d1aa03e0262f6faa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:27:43.335467 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.320643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:39919 (local address 127.0.0.1:48568)
0619 22:27:43.321852 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.321869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.321952 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.330854 (+ 8902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:43.330858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:43.330875 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:27:43.330989 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.330994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.332791 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.332795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.333902 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.333915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.335016 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.335020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.335127 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.335155 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:27:43.335210 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 22:27:43.335299 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":125,"threads_started":1}
I20260619 22:27:43.336198 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.320775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.5.176.194:39919)
0619 22:27:43.330373 (+ 9598us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.330390 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.330415 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:27:43.330490 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.330495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.330690 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.331142 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:27:43.331310 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.332659 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.334063 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.334873 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.335769 (+ 896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.335920 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.335970 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.336066 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9471,"thread_start_us":71,"threads_started":1}
I20260619 22:27:43.339311 17355 tablet_service.cc:1467] Processing CreateTablet for tablet c2d271f2460947d1aa03e0262f6faa14 (DEFAULT_TABLE table=test-table [id=800e13d498e44ac7b11b9dfe498d7b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:27:43.340468 17355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d271f2460947d1aa03e0262f6faa14. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 22:27:43 dist-test-slave-bt27 krb5kdc[16595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:27:43.351032 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.321414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:38451 (local address 127.0.0.1:36460)
0619 22:27:43.322206 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.322227 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.322309 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.323142 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:43.323145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:43.323161 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:27:43.323246 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.323251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.325398 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.325402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.326952 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.326963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.350533 (+ 23570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.350538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.350678 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.350715 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 22:27:43.350785 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 22:27:43.350873 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":78,"threads_started":1}
I20260619 22:27:43.354897 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.321565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.5.176.193:38451)
0619 22:27:43.322543 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.322550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.322573 (+ 23us) server_negotiation.cc:407] Connection header received
0619 22:27:43.322642 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.322647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.322838 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.323005 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:27:43.323384 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.325250 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.327094 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.327912 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.351374 (+ 23462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.354542 (+ 3168us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.354607 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.354726 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":113,"threads_started":1}
I20260619 22:27:43.356796 17575 tablet_bootstrap.cc:492] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Bootstrap starting.
I20260619 22:27:43.357975 17217 tablet_service.cc:1467] Processing CreateTablet for tablet c2d271f2460947d1aa03e0262f6faa14 (DEFAULT_TABLE table=test-table [id=800e13d498e44ac7b11b9dfe498d7b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:27:43.359176 17217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d271f2460947d1aa03e0262f6faa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:27:43.373311 17576 tablet_bootstrap.cc:492] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: Bootstrap starting.
I20260619 22:27:43.376250 17576 tablet_bootstrap.cc:654] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: Neither blocks nor log segments found. Creating new log.
I20260619 22:27:43.377195 17576 log.cc:824] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: Log is configured to *not* fsync() on all Append() calls
I20260619 22:27:43.377978 17575 tablet_bootstrap.cc:654] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Neither blocks nor log segments found. Creating new log.
I20260619 22:27:43.378562 17577 tablet_bootstrap.cc:492] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: Bootstrap starting.
I20260619 22:27:43.378887 17575 log.cc:824] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Log is configured to *not* fsync() on all Append() calls
I20260619 22:27:43.381031 17577 tablet_bootstrap.cc:654] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: Neither blocks nor log segments found. Creating new log.
I20260619 22:27:43.381619 17575 tablet_bootstrap.cc:492] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: No bootstrap required, opened a new log
I20260619 22:27:43.381850 17575 ts_tablet_manager.cc:1397] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20260619 22:27:43.381831 17577 log.cc:824] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: Log is configured to *not* fsync() on all Append() calls
I20260619 22:27:43.385964 17575 raft_consensus.cc:348] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.386296 17575 raft_consensus.cc:374] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:27:43.386350 17575 raft_consensus.cc:729] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8bc7aca4a649b0b63fb8d7ec65fa58, State: Initialized, Role: FOLLOWER
I20260619 22:27:43.388106 17576 tablet_bootstrap.cc:492] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: No bootstrap required, opened a new log
I20260619 22:27:43.388330 17576 ts_tablet_manager.cc:1397] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260619 22:27:43.392899 17576 raft_consensus.cc:348] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.393215 17576 raft_consensus.cc:374] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:27:43.393268 17576 raft_consensus.cc:729] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1697a794fdf9428f8829fb9dd323cb55, State: Initialized, Role: FOLLOWER
I20260619 22:27:43.393766 17576 consensus_queue.cc:260] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.394460 17575 consensus_queue.cc:260] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.395568 17575 ts_tablet_manager.cc:1428] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260619 22:27:43.396174 17577 tablet_bootstrap.cc:492] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: No bootstrap required, opened a new log
I20260619 22:27:43.396402 17577 ts_tablet_manager.cc:1397] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20260619 22:27:43.396461 17576 ts_tablet_manager.cc:1428] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 22:27:43.396785 17421 heartbeater.cc:503] Master 127.5.176.254:40015 was elected leader, sending a full tablet report...
I20260619 22:27:43.396858 17555 heartbeater.cc:503] Master 127.5.176.254:40015 was elected leader, sending a full tablet report...
I20260619 22:27:43.401849 17577 raft_consensus.cc:348] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.402120 17577 raft_consensus.cc:374] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:27:43.402201 17577 raft_consensus.cc:729] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19be37eca15440d5a9dd3d283e5c69d9, State: Initialized, Role: FOLLOWER
I20260619 22:27:43.402644 17577 consensus_queue.cc:260] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.403604 17577 ts_tablet_manager.cc:1428] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 22:27:43.403915 17283 heartbeater.cc:503] Master 127.5.176.254:40015 was elected leader, sending a full tablet report...
W20260619 22:27:43.468555 17284 tablet.cc:2369] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:27:43.471015 17556 tablet.cc:2369] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:27:43.516225 17581 raft_consensus.cc:482] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:27:43.516497 17581 raft_consensus.cc:504] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.517925 17581 leader_election.cc:288] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8bc7aca4a649b0b63fb8d7ec65fa58 (127.5.176.195:42555), 19be37eca15440d5a9dd3d283e5c69d9 (127.5.176.193:38451)
I20260619 22:27:43.532684 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.519275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:36837 (local address 127.5.176.195:42555)
0619 22:27:43.519542 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.519547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.524612 (+ 5065us) server_negotiation.cc:407] Connection header received
0619 22:27:43.524867 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.524871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.524920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.524970 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:27:43.525696 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.526542 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.531287 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.531934 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.532035 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.532091 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.532444 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.532532 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260619 22:27:43.533180 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.518346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42555 (local address 127.5.176.194:36837)
0619 22:27:43.524463 (+ 6117us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.524482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.524716 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.525113 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:43.525115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:43.525133 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:27:43.525274 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.525281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.526710 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.526714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.528127 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.528138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.532198 (+ 4060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.532202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.532300 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.532323 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:27:43.532934 (+ 611us) client_negotiation.cc:240] Negotiation successful
0619 22:27:43.533017 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6011,"thread_start_us":127,"threads_started":1}
I20260619 22:27:43.533874 17509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d271f2460947d1aa03e0262f6faa14" candidate_uuid: "1697a794fdf9428f8829fb9dd323cb55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" is_pre_election: true
I20260619 22:27:43.534247 17509 raft_consensus.cc:2454] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1697a794fdf9428f8829fb9dd323cb55 in term 0.
I20260619 22:27:43.534833 17309 leader_election.cc:302] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1697a794fdf9428f8829fb9dd323cb55, ba8bc7aca4a649b0b63fb8d7ec65fa58; no voters:
I20260619 22:27:43.535178 17581 raft_consensus.cc:2781] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:27:43.535271 17581 raft_consensus.cc:482] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:27:43.535326 17581 raft_consensus.cc:3037] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:27:43.536672 17581 raft_consensus.cc:504] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.537083 17581 leader_election.cc:288] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [CANDIDATE]: Term 1 election: Requested vote from peers ba8bc7aca4a649b0b63fb8d7ec65fa58 (127.5.176.195:42555), 19be37eca15440d5a9dd3d283e5c69d9 (127.5.176.193:38451)
I20260619 22:27:43.537571 17509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d271f2460947d1aa03e0262f6faa14" candidate_uuid: "1697a794fdf9428f8829fb9dd323cb55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58"
I20260619 22:27:43.537704 17509 raft_consensus.cc:3037] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:27:43.539038 17509 raft_consensus.cc:2454] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1697a794fdf9428f8829fb9dd323cb55 in term 1.
I20260619 22:27:43.539373 17309 leader_election.cc:302] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1697a794fdf9428f8829fb9dd323cb55, ba8bc7aca4a649b0b63fb8d7ec65fa58; no voters:
I20260619 22:27:43.539543 17581 raft_consensus.cc:2781] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:27:43.539749 17581 raft_consensus.cc:686] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [term 1 LEADER]: Becoming Leader. State: Replica: 1697a794fdf9428f8829fb9dd323cb55, State: Running, Role: LEADER
I20260619 22:27:43.540126 17581 consensus_queue.cc:237] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } }
I20260619 22:27:43.543489 17070 catalog_manager.cc:5495] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1697a794fdf9428f8829fb9dd323cb55 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "1697a794fdf9428f8829fb9dd323cb55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1697a794fdf9428f8829fb9dd323cb55" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 39919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20260619 22:27:43.548758 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.519395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:33611 (local address 127.5.176.193:38451)
0619 22:27:43.519638 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.519642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.525620 (+ 5978us) server_negotiation.cc:407] Connection header received
0619 22:27:43.528401 (+ 2781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.528405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.528460 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.528509 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:27:43.528920 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.529695 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.547197 (+ 17502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.547913 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.548026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.548080 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.548523 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.548618 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260619 22:27:43.549238 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.518789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:38451 (local address 127.5.176.194:33611)
0619 22:27:43.525539 (+ 6750us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.525552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.528271 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.528636 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:27:43.528639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:27:43.528655 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:27:43.528787 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.528793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.529850 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.529853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.531131 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:27:43.531142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.548193 (+ 17051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.548197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:27:43.548320 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.548344 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:27:43.548995 (+ 651us) client_negotiation.cc:240] Negotiation successful
0619 22:27:43.549088 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6685,"thread_start_us":97,"threads_started":1}
I20260619 22:27:43.549875 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d271f2460947d1aa03e0262f6faa14" candidate_uuid: "1697a794fdf9428f8829fb9dd323cb55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19be37eca15440d5a9dd3d283e5c69d9" is_pre_election: true
I20260619 22:27:43.550235 17222 raft_consensus.cc:2454] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1697a794fdf9428f8829fb9dd323cb55 in term 0.
I20260619 22:27:43.550597 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d271f2460947d1aa03e0262f6faa14" candidate_uuid: "1697a794fdf9428f8829fb9dd323cb55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19be37eca15440d5a9dd3d283e5c69d9"
I20260619 22:27:43.550733 17235 raft_consensus.cc:3037] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:27:43.552141 17235 raft_consensus.cc:2454] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1697a794fdf9428f8829fb9dd323cb55 in term 1.
W20260619 22:27:43.566869 17422 tablet.cc:2369] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:27:43.616534 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.611379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.5.176.194:39919)
0619 22:27:43.611528 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:27:43.611533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:27:43.611552 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:27:43.611644 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:27:43.611647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:27:43.611715 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:27:43.611834 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:27:43.612225 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.613056 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.614355 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:27:43.614688 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:27:43.614793 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:27:43.615243 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:27:43.615839 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:27:43.615962 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:27:43.616305 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 22:27:43.616383 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:27:43.624967 17509 raft_consensus.cc:1261] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 1 FOLLOWER]: Refusing update from remote peer 1697a794fdf9428f8829fb9dd323cb55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:27:43.626039 17235 raft_consensus.cc:1261] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 1 FOLLOWER]: Refusing update from remote peer 1697a794fdf9428f8829fb9dd323cb55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:27:43.626564 17587 consensus_queue.cc:1035] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8bc7aca4a649b0b63fb8d7ec65fa58" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:27:43.627839 17587 consensus_queue.cc:1035] T c2d271f2460947d1aa03e0262f6faa14 P 1697a794fdf9428f8829fb9dd323cb55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19be37eca15440d5a9dd3d283e5c69d9" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:27:43.634953 17592 mvcc.cc:205] Tried to move back new op lower bound from 7298695428598935552 to 7298695428263636992. Current Snapshot: MvccSnapshot[applied={T|T < 7298695428598935552}]
I20260619 22:27:43.636386 17594 mvcc.cc:205] Tried to move back new op lower bound from 7298695428598935552 to 7298695428263636992. Current Snapshot: MvccSnapshot[applied={T|T < 7298695428598935552}]
I20260619 22:27:43.648581 17595 mvcc.cc:205] Tried to move back new op lower bound from 7298695428598935552 to 7298695428263636992. Current Snapshot: MvccSnapshot[applied={T|T < 7298695428598935552}]
I20260619 22:27:43.656661 17070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:27:43.656877 17070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:27:43.659530 17070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 945fe8adf96f47bbb87eb2794e0bdcf9: Sending DeleteTablet for 3 replicas of tablet c2d271f2460947d1aa03e0262f6faa14
I20260619 22:27:43.660705 17355 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d271f2460947d1aa03e0262f6faa14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:27:43 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:48568
I20260619 22:27:43.660945 17217 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d271f2460947d1aa03e0262f6faa14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:27:43 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:36460
I20260619 22:27:43.661008 17487 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d271f2460947d1aa03e0262f6faa14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:27:43 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:39586
I20260619 22:27:43.661403 17608 tablet_replica.cc:330] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9: stopping tablet replica
I20260619 22:27:43.661810 17608 raft_consensus.cc:2229] T c2d271f2460947d1aa03e0262f6faa14 P 19be37eca15440d5a9dd3d283e5c69d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:27:43.662115 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 17152
I20260619 22:27:43.670418 17609 tablet_replica.cc:330] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: stopping tablet replica
I20260619 22:27:43.670862 17609 raft_consensus.cc:2229] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 22:27:43.671124 17057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:38451 (error 108)
I20260619 22:27:43.671250 17609 raft_consensus.cc:2258] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 22:27:43.672427 17571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:27:43.671921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:38451 (local address 127.0.0.1:36466)
0619 22:27:43.672050 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.672068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:27:43.672161 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:27:43.672282 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:38451: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:27:43.672672 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 17291
I20260619 22:27:43.673568 17609 ts_tablet_manager.cc:1905] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 22:27:43.674474 17057 catalog_manager.cc:4584] TS 19be37eca15440d5a9dd3d283e5c69d9 (127.5.176.193:38451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2d271f2460947d1aa03e0262f6faa14: Network error: Client connection negotiation failed: client connection to 127.5.176.193:38451: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260619 22:27:43.676544 17609 ts_tablet_manager.cc:1918] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:27:43.676651 17609 log.cc:1196] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/c2d271f2460947d1aa03e0262f6faa14
I20260619 22:27:43.676981 17609 ts_tablet_manager.cc:1939] T c2d271f2460947d1aa03e0262f6faa14 P ba8bc7aca4a649b0b63fb8d7ec65fa58: Deleting consensus metadata
I20260619 22:27:43.678471 17056 catalog_manager.cc:4841] TS ba8bc7aca4a649b0b63fb8d7ec65fa58 (127.5.176.195:42555): tablet c2d271f2460947d1aa03e0262f6faa14 (table test-table [id=800e13d498e44ac7b11b9dfe498d7b20]) successfully deleted
I20260619 22:27:43.681739 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:27:43.681418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:39919 (local address 127.0.0.1:48582)
0619 22:27:43.681552 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 22:27:43.681642 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 22:27:43.681787 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 17426
I20260619 22:27:43.687001 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 17039
I20260619 22:27:43.694100 5827 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 22:27:44.029783 5827 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 22:27:44.029932 5827 mini_ranger.cc:66] Stopping Ranger...
I20260619 22:27:44.376825 5827 mini_ranger.cc:68] Stopped Ranger
2026-06-19 22:27:44.377 UTC [16625] LOG: received smart shutdown request
2026-06-19 22:27:44.380 UTC [16625] LOG: background worker "logical replication launcher" (PID 16634) exited with exit code 1
2026-06-19 22:27:44.381 UTC [16629] LOG: shutting down
2026-06-19 22:27:44.557 UTC [16625] LOG: database system is shut down
2026-06-19T22:27:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46563 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:27:44 dist-test-slave-bt27 krb5kdc[17619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:27:44 dist-test-slave-bt27 krb5kdc[17619](info): set up 2 sockets
Jun 19 22:27:44 dist-test-slave-bt27 krb5kdc[17619](info): commencing operation
krb5kdc: starting...
W20260619 22:27:45.722458 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:27:45 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:27:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/rangeradmin_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/rangeradmin_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/rangerlookup_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/rangerlookup_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
I20260619 22:27:45.892477 5827 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4QvkS_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/postgres -l logfile start
2026-06-19 22:27:48.342 UTC [17649] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 22:27:48.342 UTC [17649] LOG: listening on IPv4 address "127.5.176.212", port 35969
2026-06-19 22:27:48.345 UTC [17649] LOG: listening on Unix socket "/tmp/.s.PGSQL.35969"
2026-06-19 22:27:48.366 UTC [17652] LOG: database system was shut down at 2026-06-19 22:27:46 UTC
2026-06-19 22:27:48.370 UTC [17649] LOG: database system is ready to accept connections
I20260619 22:27:49.325995 5827 mini_postgres.cc:96] Postgres bound to 35969
2026-06-19 22:27:49.332 UTC [17660] FATAL: database "slave" does not exist
127.5.176.212:35969 - accepting connections
I20260619 22:27:49.333056 5827 mini_ranger.cc:161] Starting Ranger...
I20260619 22:27:49.347849 5827 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 22:27:49.786666 5827 mini_ranger.cc:87] Created ranger Postgres database
I20260619 22:27:49.786856 5827 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-admin
2026-06-19 22:27:50,074 [I] DB FLAVOR :POSTGRES
2026-06-19 22:27:50,074 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:27:50,074 [I] Checking connection..
2026-06-19 22:27:50,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 22:27:50,331 [I] Checking connection passed.
2026-06-19 22:27:50,331 [I] --------- Verifying version history table ---------
2026-06-19 22:27:50,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:27:50,575 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:27:50,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:27:50,840 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 22:27:50,840 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 22:27:50,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 22:27:51.067 UTC [17724] WARNING: there is no transaction in progress
2026-06-19 22:27:51,074 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 22:27:51,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 22:27:51,331 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 22:27:51,332 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:27:51,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 22:27:51,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 22:27:51,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bt27.c.gcp-upstream.internal','N') ;"
2026-06-19 22:27:52,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:27:52,302 [I] Table x_portal_user does not exist in database ranger
2026-06-19 22:27:52,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:27:52,565 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 22:27:52,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:27:52,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:27:53,030 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 22:27:53,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 22:27:53.996 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.013 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.029 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.044 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.060 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.077 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.135 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.142 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.156 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.167 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.178 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.193 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.204 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.211 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.218 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54.225 UTC [17832] WARNING: there is no transaction in progress
2026-06-19 22:27:54,839 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 22:27:54,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 22:27:55,099 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 22:27:55,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 22:27:55,351 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 22:27:55,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 22:27:55,596 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 22:27:55,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 22:27:55,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 22:27:56,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bt27.c.gcp-upstream.internal';"
2026-06-19 22:27:56,328 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 22:27:56,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 22:27:56,595 [I] DB_PATCHES have already been applied
I20260619 22:27:56.601477 5827 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-admin:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4QvkS_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 22:27:56.601547 5827 mini_ranger.cc:193] Using host: 127.5.176.212
I20260619 22:27:56.605481 5827 mini_ranger.cc:227] Ranger admin URL: http://127.5.176.212:60887
Jun 19, 2026 10:27:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:27:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:27:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:27:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:27:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:27:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:27:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 22:27:57.719672 5827 mini_ranger.cc:160] Time spent starting Ranger: real 8.387s user 0.000s sys 0.010s
Jun 19, 2026 10:28:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:28:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 10:28:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:28:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:28:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 22:28:19.069423 5827 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/rangerkms_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/rangerkms_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.176.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.176.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.176.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/HTTP_127.5.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 22:28:19 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908099, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 22:28:19.152211 5827 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 22:28:19.161988 5827 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 22:28:19.974591 5827 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 22:28:19.974675 5827 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms
I20260619 22:28:19.974699 5827 mini_ranger_kms.cc:227] Using postgres at 127.5.176.212:35969
2026-06-19 22:28:20,153 [I] DB FLAVOR :POSTGRES
2026-06-19 22:28:20,153 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 22:28:20,153 [I] Checking connection
2026-06-19 22:28:20,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 22:28:20,400 [I] connection success
2026-06-19 22:28:20,400 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 22:28:20,401 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 22:28:20,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 22:28:20,649 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 22:28:20,649 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 22:28:20,649 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 22:28:20,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.176.212:35969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 22:28:20,903 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 22:28:21.383502 5827 mini_ranger_kms.cc:310] Created kms service
I20260619 22:28:21.596154 5827 mini_ranger_kms.cc:326] Created kudu user
I20260619 22:28:21.650866 5827 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 22:28:21 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908099, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:22.045594 5827 mini_ranger_kms.cc:384] Added ranger policy
I20260619 22:28:22.045758 5827 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms:/tmp/dist-test-task4QvkS_/build/debug/bin/postgresql.jar:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4QvkS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 22:28:22.045778 5827 mini_ranger_kms.cc:240] Using host: 127.5.176.212
I20260619 22:28:22.050069 5827 mini_ranger_kms.cc:276] Ranger KMS PID: 18012
I20260619 22:28:22.050171 5827 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.176.212:44043
Jun 19, 2026 10:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 22:28:24.346642 5827 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.194s user 0.001s sys 0.162s
I20260619 22:28:24.346774 5827 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 22:28:24.346856 5827 mini_ranger_kms.cc:401] 127.5.176.212:44043/kms/v1/keys
Jun 19 22:28:24 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.176.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 22:28:24 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.176.212@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
Jun 19 22:28:25 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.176.212@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
Jun 19 22:28:27 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:28:27.585440 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34559
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:34559
--ranger_config_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:27.701875 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:27.702217 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:27.702261 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:27.706518 18063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:27.706588 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:27.706609 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:27.706625 18063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:27.706641 18063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:27.712460 18063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.176.212:44043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:34559
--ranger_config_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:27.713696 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:27.714824 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:27.724027 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:27.724027 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:27.724206 18063 server_base.cc:1034] running on GCE node
W20260619 22:28:27.724534 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:27.725308 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:27.726883 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:27.728148 18063 hybrid_clock.cc:648] HybridClock initialized: now 1781908107728132 us; error 34 us; skew 500 ppm
Jun 19 22:28:27 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:27.731817 18063 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:27.732951 18063 webserver.cc:460] Webserver started at http://127.5.176.254:45409/ using document root <none> and password file <none>
I20260619 22:28:27.733662 18063 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:27.733722 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:27.733901 18063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:28:27 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:27.780535 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "8a4756fcf07547d9a99699269f203ba7"
format_stamp: "Formatted at 2026-06-19 22:28:27 on dist-test-slave-bt27"
server_key: "c3ede9d99666a6446c76833092de2a9c"
server_key_iv: "3e65ae6690742baefada7d4f8e72f3fc"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:27.781167 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "8a4756fcf07547d9a99699269f203ba7"
format_stamp: "Formatted at 2026-06-19 22:28:27 on dist-test-slave-bt27"
server_key: "c3ede9d99666a6446c76833092de2a9c"
server_key_iv: "3e65ae6690742baefada7d4f8e72f3fc"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:27.785166 18063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 22:28:27.799656 18078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:27.801105 18063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:28:27.801221 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "8a4756fcf07547d9a99699269f203ba7"
format_stamp: "Formatted at 2026-06-19 22:28:27 on dist-test-slave-bt27"
server_key: "c3ede9d99666a6446c76833092de2a9c"
server_key_iv: "3e65ae6690742baefada7d4f8e72f3fc"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:27.801394 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:27.825836 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:27.829105 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:27.829324 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:27.837062 18063 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:34559
I20260619 22:28:27.837075 18130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:34559 every 8 connection(s)
I20260619 22:28:27.838037 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:27.841073 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:27.845715 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18063
I20260619 22:28:27.845865 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:28:27.846197 5827 external_mini_cluster.cc:1428] Setting key e9c7c3f3bc4c8c6e465ca91ab8f400b6
I20260619 22:28:27.847661 18131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Bootstrap starting.
I20260619 22:28:27.850310 18131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:27.851142 18131 log.cc:824] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:27.853168 18131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: No bootstrap required, opened a new log
Jun 19 22:28:27 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:27.858733 18131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:27.859087 18131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:27.859164 18131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4756fcf07547d9a99699269f203ba7, State: Initialized, Role: FOLLOWER
I20260619 22:28:27.859612 18131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:27.859802 18131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:27.859867 18131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:27.859983 18131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:27.861336 18131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:27.861733 18131 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4756fcf07547d9a99699269f203ba7; no voters:
I20260619 22:28:27.862099 18131 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:28:27.862246 18136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:27.862502 18136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 LEADER]: Becoming Leader. State: Replica: 8a4756fcf07547d9a99699269f203ba7, State: Running, Role: LEADER
I20260619 22:28:27.862567 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:27.847692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.5.176.254:34559)
0619 22:28:27.848145 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:27.848163 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:27.848200 (+ 37us) server_negotiation.cc:407] Connection header received
0619 22:28:27.848868 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:27.848885 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:27.849222 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:27.849503 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:27.850560 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:27.851595 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:27.852502 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:27.852976 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:27.853149 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:27.856235 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:27.856243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:27.856265 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:27.856303 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:27.858396 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:27.858886 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:27.858892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:27.858899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:27.859040 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:27.859721 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:27.859724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:27.859727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:27.860602 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:27.860826 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:27.861080 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 22:28:27.861415 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260619 22:28:27.863236 18136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:27.863533 18131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:27.865024 18138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a4756fcf07547d9a99699269f203ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } } }
I20260619 22:28:27.865187 18138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:27.865025 18137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4756fcf07547d9a99699269f203ba7. Latest consensus state: current_term: 1 leader_uuid: "8a4756fcf07547d9a99699269f203ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } } }
I20260619 22:28:27.865355 18137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:27.865962 18131 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 22:28:27.866375 18147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:27.872538 18147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:27.880457 18147 catalog_manager.cc:1347] Generated new cluster ID: 6387e33fbf6d4d578ee27d4f673c7cb3
I20260619 22:28:27.880537 18147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:27.896986 18147 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:28:27.898589 18147 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:27.905683 18147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Generated new TSK 0
I20260619 22:28:27.906596 18147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:28:28.575484 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34559
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:28.712157 18176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:28.712574 18176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:28.712648 18176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:28.719004 18176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:28.719102 18176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:28.719198 18176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:28:28.727694 18176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.176.212:44043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:28.729168 18176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:28.730659 18176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:28.743386 18183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:28.743498 18186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:28.743563 18184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:28.743748 18176 server_base.cc:1034] running on GCE node
I20260619 22:28:28.744655 18176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:28.745497 18176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:28.746682 18176 hybrid_clock.cc:648] HybridClock initialized: now 1781908108746651 us; error 45 us; skew 500 ppm
Jun 19 22:28:28 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:28.750711 18176 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:28:28.751803 18176 webserver.cc:460] Webserver started at http://127.5.176.193:37583/ using document root <none> and password file <none>
I20260619 22:28:28.752538 18176 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:28.752668 18176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:28.752950 18176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:28:28 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:28.775924 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "de3348ea5494400bb0c4388d098f90e6"
format_stamp: "Formatted at 2026-06-19 22:28:28 on dist-test-slave-bt27"
server_key: "bc3d1ed13914755e5c063f7235048458"
server_key_iv: "ebfc3abf6eb27b64ffff98e1d388ccdf"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:28.776707 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "de3348ea5494400bb0c4388d098f90e6"
format_stamp: "Formatted at 2026-06-19 22:28:28 on dist-test-slave-bt27"
server_key: "bc3d1ed13914755e5c063f7235048458"
server_key_iv: "ebfc3abf6eb27b64ffff98e1d388ccdf"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:28.781688 18176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 22:28:28.801067 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:28.802807 18176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:28:28.802944 18176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "de3348ea5494400bb0c4388d098f90e6"
format_stamp: "Formatted at 2026-06-19 22:28:28 on dist-test-slave-bt27"
server_key: "bc3d1ed13914755e5c063f7235048458"
server_key_iv: "ebfc3abf6eb27b64ffff98e1d388ccdf"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:28.803076 18176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:28.823590 18176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:28.831492 18176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:28.831780 18176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:28.832499 18176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:28.834003 18176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:28.834105 18176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:28.834232 18176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:28.834291 18176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:28.844709 18176 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:44015
I20260619 22:28:28.845904 18176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:28:28.855984 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18176
I20260619 22:28:28.856122 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:28:28.856413 5827 external_mini_cluster.cc:1428] Setting key 961734fb133e5f74762c15581f2eae72
I20260619 22:28:28.859390 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:44015 every 8 connection(s)
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Jun 19 22:28:28 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:28:28.877357 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:28.854552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34559 (local address 127.5.176.193:55613)
0619 22:28:28.859910 (+ 5358us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:28.859973 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:28.861082 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:28.863516 (+ 2434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:28.863527 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:28.863918 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:28.864731 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:28.864757 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:28.866048 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:28.866052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:28.867540 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:28.867553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:28.868294 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:28.868299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:28.868525 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:28.869315 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:28.869338 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:28.871028 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:28.873582 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:28.873590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:28.873610 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:28.874018 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:28.874415 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:28.874420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:28.874422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:28.874636 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:28.875175 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:28.875183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:28.875565 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 22:28:28.875784 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 22:28:28.876191 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5097,"thread_start_us":151,"threads_started":1}
I20260619 22:28:28.878881 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:28.856869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55613 (local address 127.5.176.254:34559)
0619 22:28:28.862967 (+ 6098us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:28.862978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:28.862995 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:28:28.863046 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:28.863049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:28.863123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:28.864914 (+ 1791us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:28.865050 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:28.865889 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:28.867718 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:28.868083 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:28.870161 (+ 2078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:28.871310 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:28.871314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:28.871317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:28.871346 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:28.873418 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:28.874203 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:28.874206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:28.874208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:28.874301 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:28.874771 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:28.874774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:28.874776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:28.875022 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:28.878467 (+ 3445us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:28.878563 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 22:28:28.878713 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6000,"thread_start_us":131,"threads_started":1}
I20260619 22:28:28.880385 18309 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34559
I20260619 22:28:28.880653 18309 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:28.881299 18309 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:28.883261 18095 ts_manager.cc:194] Registered new tserver with Master: de3348ea5494400bb0c4388d098f90e6 (127.5.176.193:44015)
I20260619 22:28:28.884979 18095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:55613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:28:28.893011 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34559
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:28:28 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 22:28:29.051074 18315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:29.051406 18315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:29.051502 18315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:29.057011 18315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:29.057123 18315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:29.057229 18315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:28:29.064081 18315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.176.212:44043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:29.066031 18315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:29.067391 18315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:29.079715 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:29.079996 18315 server_base.cc:1034] running on GCE node
W20260619 22:28:29.079715 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:29.080214 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:29.080842 18315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:29.081650 18315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:29.082865 18315 hybrid_clock.cc:648] HybridClock initialized: now 1781908109082827 us; error 58 us; skew 500 ppm
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:29.085839 18315 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:28:29.086913 18315 webserver.cc:460] Webserver started at http://127.5.176.194:37419/ using document root <none> and password file <none>
I20260619 22:28:29.087560 18315 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:29.087656 18315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:29.087906 18315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:29.109935 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "066b24781aca4da8a4cee47ef7a62670"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "673b2ec53f6e1c26875fa811e6c25777"
server_key_iv: "858a17b682185f01ba21f12fd297fdb9"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.110786 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "066b24781aca4da8a4cee47ef7a62670"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "673b2ec53f6e1c26875fa811e6c25777"
server_key_iv: "858a17b682185f01ba21f12fd297fdb9"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.116135 18315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:28:29.137619 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.139511 18315 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260619 22:28:29.139654 18315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "066b24781aca4da8a4cee47ef7a62670"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "673b2ec53f6e1c26875fa811e6c25777"
server_key_iv: "858a17b682185f01ba21f12fd297fdb9"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.139801 18315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:29.163822 18315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:29.167814 18315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:29.168030 18315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:29.168792 18315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:29.169826 18315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:29.169890 18315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.169943 18315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:29.169976 18315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.180941 18315 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:46845
I20260619 22:28:29.180956 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:46845 every 8 connection(s)
I20260619 22:28:29.182183 18315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:28:29.186367 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18315
I20260619 22:28:29.186508 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:28:29.186817 5827 external_mini_cluster.cc:1428] Setting key 4d1104ef1544360cad75823bcce87d5d
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:29.200511 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:29.184386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:55971 (local address 127.5.176.254:34559)
0619 22:28:29.184565 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:29.184571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:29.185722 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 22:28:29.187388 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:29.187394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:29.187480 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:29.187548 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:29.189511 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.190455 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.192105 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.192487 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.192566 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:29.195801 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:29.195806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:29.195809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:29.195839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:29.197741 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:29.198502 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:29.198506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:29.198507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:29.198602 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:29.199061 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:29.199065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:29.199067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:29.199305 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:29.199390 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:29.200134 (+ 744us) server_negotiation.cc:299] Negotiation successful
0619 22:28:29.200287 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:28:29.202589 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:29.184879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34559 (local address 127.5.176.194:55971)
0619 22:28:29.185519 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:29.185591 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:29.187017 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:29.188048 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:29.188062 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:29.188482 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:29.189286 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:29.189311 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.190643 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.190648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:29.191964 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:29.191976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.192791 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.192795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:29.193005 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:29.193729 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:29.193750 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:29.195578 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:29.197898 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:29.197905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:29.197925 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:29.198360 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:29.198729 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:29.198732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:29.198734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:29.198941 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:29.199476 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:29.199484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:29.199839 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 22:28:29.201011 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0619 22:28:29.201393 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":158,"threads_started":1}
I20260619 22:28:29.204438 18445 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34559
I20260619 22:28:29.204777 18445 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:29.205509 18445 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:29.206821 18095 ts_manager.cc:194] Registered new tserver with Master: 066b24781aca4da8a4cee47ef7a62670 (127.5.176.194:46845)
I20260619 22:28:29.207635 18095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:55971
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:28:29.261466 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34559
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
W20260619 22:28:29.423693 18450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:29.424134 18450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:29.424253 18450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:29.429549 18450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:29.429666 18450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:29.429785 18450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:28:29.435652 18450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.176.212:44043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:34559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:29.437079 18450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:29.438232 18450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:29.448215 18456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:29.448324 18458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:29.448215 18455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:29.448863 18450 server_base.cc:1034] running on GCE node
I20260619 22:28:29.449258 18450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:29.449927 18450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:29.451198 18450 hybrid_clock.cc:648] HybridClock initialized: now 1781908109451162 us; error 50 us; skew 500 ppm
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:29.454236 18450 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:28:29.455482 18450 webserver.cc:460] Webserver started at http://127.5.176.195:45785/ using document root <none> and password file <none>
I20260619 22:28:29.456169 18450 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:29.456266 18450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:29.456517 18450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:29.473424 18450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "2d18a47331c14e489f6a7f29aa7b0f6c"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "86e9bea5f17b4ed1f4aa9f3512a0cc49"
server_key_iv: "171d56b62e2af1afab3871e88776e977"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.474182 18450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "2d18a47331c14e489f6a7f29aa7b0f6c"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "86e9bea5f17b4ed1f4aa9f3512a0cc49"
server_key_iv: "171d56b62e2af1afab3871e88776e977"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.483201 18450 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.001s
I20260619 22:28:29.505004 18465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.506790 18450 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:28:29.507020 18450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "2d18a47331c14e489f6a7f29aa7b0f6c"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "86e9bea5f17b4ed1f4aa9f3512a0cc49"
server_key_iv: "171d56b62e2af1afab3871e88776e977"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.507265 18450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:29.529394 18450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:29.532634 18450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:29.532848 18450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:29.533479 18450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:29.534555 18450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:29.534621 18450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.534713 18450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:29.534747 18450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.545455 18450 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:40359
I20260619 22:28:29.545732 18578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:40359 every 8 connection(s)
I20260619 22:28:29.546736 18450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:28:29.554458 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18450
I20260619 22:28:29.554586 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:28:29.554895 5827 external_mini_cluster.cc:1428] Setting key acc3948fdb5164fbde80b51f388ae663
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:29.561065 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:29.548651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:37107 (local address 127.5.176.254:34559)
0619 22:28:29.548779 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:29.548785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:29.549712 (+ 927us) server_negotiation.cc:407] Connection header received
0619 22:28:29.550829 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:29.550834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:29.550901 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:29.550958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:29.552390 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.552995 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.554106 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.554473 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.554582 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:29.557486 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:29.557491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:29.557499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:29.557523 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:29.559029 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:29.559552 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:29.559557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:29.559559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:29.559640 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:29.559993 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:29.559997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:29.559998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:29.560186 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:29.560292 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:29.560788 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 22:28:29.560906 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:28:29.561995 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:29.549096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34559 (local address 127.5.176.195:37107)
0619 22:28:29.549559 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:29.549610 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:29.550566 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:29.551423 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:29.551444 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:29.551768 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:29.552241 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:29.552254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.553118 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.553128 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:29.553978 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:29.553987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:29.554766 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:29.554770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:29.554982 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:29.555640 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:29.555662 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:29.557323 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:29.559163 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:29.559169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:29.559186 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:29.559450 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:29.559738 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:29.559741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:29.559743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:29.559890 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:29.560290 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:29.560297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:29.560558 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 22:28:29.560752 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 22:28:29.561078 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
I20260619 22:28:29.563122 18579 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34559
I20260619 22:28:29.563377 18579 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:29.563869 18579 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:29.564999 18095 ts_manager.cc:194] Registered new tserver with Master: 2d18a47331c14e489f6a7f29aa7b0f6c (127.5.176.195:40359)
I20260619 22:28:29.565640 18095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:37107
I20260619 22:28:29.568918 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:28:29.569603 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18176
I20260619 22:28:29.575685 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18315
I20260619 22:28:29.582175 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18450
I20260619 22:28:29.589421 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18063
I20260619 22:28:29.595265 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34559
--webserver_interface=127.5.176.254
--webserver_port=45409
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:34559
--ranger_config_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:29.710440 18582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:29.710734 18582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:29.710822 18582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:29.714979 18582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:29.715041 18582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:29.715060 18582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:29.715078 18582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:29.715147 18582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:29.720685 18582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:34559
--ranger_config_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=45409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:29.721809 18582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:29.722931 18582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:29.731519 18590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:29.731523 18588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:29.731519 18587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:29.731799 18582 server_base.cc:1034] running on GCE node
I20260619 22:28:29.732501 18582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:29.733929 18582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:29.735143 18582 hybrid_clock.cc:648] HybridClock initialized: now 1781908109735097 us; error 59 us; skew 500 ppm
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:29.738345 18582 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:29.739260 18582 webserver.cc:460] Webserver started at http://127.5.176.254:45409/ using document root <none> and password file <none>
I20260619 22:28:29.739827 18582 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:29.739921 18582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:28:29 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:29.785730 18582 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:28:29.788312 18598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:29.789640 18582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 22:28:29.789759 18582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "8a4756fcf07547d9a99699269f203ba7"
format_stamp: "Formatted at 2026-06-19 22:28:27 on dist-test-slave-bt27"
server_key: "c3ede9d99666a6446c76833092de2a9c"
server_key_iv: "3e65ae6690742baefada7d4f8e72f3fc"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:29.790388 18582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:29.806618 18582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:29.809311 18582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:29.809484 18582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:29.816864 18582 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:34559
I20260619 22:28:29.816887 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:34559 every 8 connection(s)
I20260619 22:28:29.817972 18582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:29.822269 18651 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:28:29.822968 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18582
I20260619 22:28:29.823462 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:44015
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:34559
--webserver_port=37583
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:29.824496 18651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Bootstrap starting.
I20260619 22:28:29.841980 18651 log.cc:824] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:29.849174 18651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:28:29.849618 18651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Bootstrap complete.
I20260619 22:28:29.853848 18651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:29.854337 18651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4756fcf07547d9a99699269f203ba7, State: Initialized, Role: FOLLOWER
I20260619 22:28:29.854900 18651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:29.855023 18651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:29.855114 18651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:29.855211 18651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:28:29.857177 18651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:29.857524 18651 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4756fcf07547d9a99699269f203ba7; no voters:
I20260619 22:28:29.857883 18651 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:28:29.857998 18655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:28:29.858328 18655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [term 2 LEADER]: Becoming Leader. State: Replica: 8a4756fcf07547d9a99699269f203ba7, State: Running, Role: LEADER
I20260619 22:28:29.858700 18655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } }
I20260619 22:28:29.859251 18651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:29.860224 18656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a4756fcf07547d9a99699269f203ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } } }
I20260619 22:28:29.860376 18656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:29.860631 18657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4756fcf07547d9a99699269f203ba7. Latest consensus state: current_term: 2 leader_uuid: "8a4756fcf07547d9a99699269f203ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4756fcf07547d9a99699269f203ba7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34559 } } }
I20260619 22:28:29.860723 18657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:29.861718 18651 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 22:28:29.862531 18664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:29.868806 18664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:29.869710 18664 catalog_manager.cc:1259] Loaded cluster ID: 6387e33fbf6d4d578ee27d4f673c7cb3
I20260619 22:28:29.869863 18664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:29.872838 18664 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:29.873814 18664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Loaded TSK: 0
I20260619 22:28:29.875344 18664 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:28:29.968056 18653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:29.968494 18653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:29.968583 18653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:29.974781 18653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:29.974916 18653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:29.975046 18653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:28:29.982969 18653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:44015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=37583
--tserver_master_addrs=127.5.176.254:34559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:29.984493 18653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:29.985790 18653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:28:29.999189 18653 server_base.cc:1034] running on GCE node
W20260619 22:28:29.999389 18691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:29.999754 18689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:30.001076 18688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:30.001708 18653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:30.002700 18653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:30.003925 18653 hybrid_clock.cc:648] HybridClock initialized: now 1781908110003894 us; error 52 us; skew 500 ppm
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:30.007831 18653 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:28:30.009127 18653 webserver.cc:460] Webserver started at http://127.5.176.193:37583/ using document root <none> and password file <none>
I20260619 22:28:30.009853 18653 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:30.009971 18653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:30.045491 18653 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:28:30.049160 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.050923 18653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 22:28:30.051070 18653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "de3348ea5494400bb0c4388d098f90e6"
format_stamp: "Formatted at 2026-06-19 22:28:28 on dist-test-slave-bt27"
server_key: "bc3d1ed13914755e5c063f7235048458"
server_key_iv: "ebfc3abf6eb27b64ffff98e1d388ccdf"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:30.051811 18653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:30.076072 18653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:30.079973 18653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:30.080238 18653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:30.080940 18653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:30.081992 18653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:30.082096 18653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.082263 18653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:30.082350 18653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.094007 18653 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:44015
I20260619 22:28:30.094251 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:44015 every 8 connection(s)
I20260619 22:28:30.095620 18653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:28:30.102872 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18653
I20260619 22:28:30.103406 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:46845
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:34559
--webserver_port=37419
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:30.117202 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.098053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34559 (local address 127.5.176.193:44089)
0619 22:28:30.098632 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:30.098687 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:30.099540 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:30.100970 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:30.100979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:30.101345 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:30.101973 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:30.101996 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.103913 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.103920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:30.105385 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:30.105397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.106656 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.106663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:30.106905 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.107648 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:30.107677 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:30.109622 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:30.112798 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:30.112809 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:30.112828 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:30.113286 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:30.113869 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:30.113875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:30.113879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:30.114097 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:30.115113 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:30.115121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:30.115470 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 22:28:30.115725 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 22:28:30.116115 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":149,"threads_started":1}
I20260619 22:28:30.117323 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.097904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:44089 (local address 127.5.176.254:34559)
0619 22:28:30.099957 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:30.099978 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:30.100014 (+ 36us) server_negotiation.cc:407] Connection header received
0619 22:28:30.100237 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:30.100246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:30.100557 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:30.100893 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:30.102237 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.103711 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.105544 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.106429 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.107373 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.109919 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:30.109929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:30.109953 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:30.109999 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:30.112599 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.113505 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.113515 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.113524 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.113722 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.114280 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.114286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.114289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.114918 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:30.115081 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:30.115908 (+ 827us) server_negotiation.cc:299] Negotiation successful
0619 22:28:30.116258 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1901,"thread_start_us":150,"threads_started":1}
I20260619 22:28:30.120862 18813 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34559
I20260619 22:28:30.121124 18813 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.121721 18813 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
W20260619 22:28:30.124200 18813 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:34559 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 22:28:30.124354 18813 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.124571 18813 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:30.125654 18813 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.125900 18813 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
W20260619 22:28:30.126868 18813 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:28:30.242278 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:30.242602 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:30.242710 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:30.247155 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:30.247265 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:30.247401 18817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:28:30.253973 18817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:46845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=37419
--tserver_master_addrs=127.5.176.254:34559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:30.255677 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:30.257135 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:30.268193 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:30.268187 18825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:30.268193 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:30.268415 18817 server_base.cc:1034] running on GCE node
I20260619 22:28:30.269081 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:30.269800 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:30.271024 18817 hybrid_clock.cc:648] HybridClock initialized: now 1781908110271003 us; error 37 us; skew 500 ppm
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:30.274536 18817 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:28:30.275599 18817 webserver.cc:460] Webserver started at http://127.5.176.194:37419/ using document root <none> and password file <none>
I20260619 22:28:30.276188 18817 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:30.276296 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:30.306056 18817 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260619 22:28:30.309506 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.311254 18817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 22:28:30.311389 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "066b24781aca4da8a4cee47ef7a62670"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "673b2ec53f6e1c26875fa811e6c25777"
server_key_iv: "858a17b682185f01ba21f12fd297fdb9"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:30.312179 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:30.335649 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:30.339500 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:30.339725 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:30.340482 18817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:30.341572 18817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:30.341639 18817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.341696 18817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:30.341730 18817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.353193 18817 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:46845
I20260619 22:28:30.355310 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:46845 every 8 connection(s)
I20260619 22:28:30.355909 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:28:30.363480 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18817
I20260619 22:28:30.363907 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.176.212:44043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:40359
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:34559
--webserver_port=45785
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:30.374429 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.358443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34559 (local address 127.5.176.194:47371)
0619 22:28:30.358913 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:30.358964 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:30.360095 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:30.360913 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:30.360922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:30.361286 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:30.361982 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:30.362002 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.363154 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.363157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:30.364621 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:30.364633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.365501 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.365504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:30.365659 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.366361 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:30.366383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:30.368466 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:30.370751 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:30.370757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:30.370775 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:30.371145 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:30.371523 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:30.371528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:30.371530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:30.371749 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:30.372304 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:30.372311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:30.372694 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 22:28:30.372916 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 22:28:30.373275 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":142,"threads_started":1}
I20260619 22:28:30.375349 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.357948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:47371 (local address 127.5.176.254:34559)
0619 22:28:30.358166 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:30.358172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:30.359091 (+ 919us) server_negotiation.cc:407] Connection header received
0619 22:28:30.360361 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:30.360365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:30.360434 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:30.360540 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:30.362190 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.362998 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.364760 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.365185 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.365309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.368723 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:30.368727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:30.368730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:30.368760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:30.370598 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.371272 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.371279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.371283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.371382 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.371871 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.371877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.371880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.372141 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:30.372244 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:30.375038 (+ 2794us) server_negotiation.cc:299] Negotiation successful
0619 22:28:30.375193 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 22:28:30.376652 18947 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34559
I20260619 22:28:30.376929 18947 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.377579 18947 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
W20260619 22:28:30.379483 18947 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:34559 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 22:28:30.379628 18947 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.379819 18947 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:30.380776 18947 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.380990 18947 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
W20260619 22:28:30.381831 18947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:28:30.532673 18951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:30.532979 18951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:30.533078 18951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:30.537544 18951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:30.537624 18951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:30.537731 18951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:28:30.544667 18951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.176.212:44043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:40359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=45785
--tserver_master_addrs=127.5.176.254:34559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:30.546190 18951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:30.547569 18951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:30.558673 18959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:30.558696 18957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:30.558796 18956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:30.559381 18951 server_base.cc:1034] running on GCE node
I20260619 22:28:30.559866 18951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:30.560621 18951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:30.561780 18951 hybrid_clock.cc:648] HybridClock initialized: now 1781908110561729 us; error 71 us; skew 500 ppm
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:30.565500 18951 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:28:30.566751 18951 webserver.cc:460] Webserver started at http://127.5.176.195:45785/ using document root <none> and password file <none>
I20260619 22:28:30.567503 18951 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:30.567610 18951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:30.588557 18951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:28:30.591423 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.593753 18951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:28:30.593958 18951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "2d18a47331c14e489f6a7f29aa7b0f6c"
format_stamp: "Formatted at 2026-06-19 22:28:29 on dist-test-slave-bt27"
server_key: "86e9bea5f17b4ed1f4aa9f3512a0cc49"
server_key_iv: "171d56b62e2af1afab3871e88776e977"
server_key_version: "kuduclusterkey@0"
I20260619 22:28:30.594735 18951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:30.607627 18951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:30.611290 18951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:30.611537 18951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:30.612210 18951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:30.613250 18951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:30.613301 18951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.613354 18951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:30.613375 18951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:30.628757 18951 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:40359
I20260619 22:28:30.628903 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:40359 every 8 connection(s)
I20260619 22:28:30.629961 18951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
Jun 19 22:28:30 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:30.640215 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 18951
I20260619 22:28:30.646559 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.631983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:48871 (local address 127.5.176.254:34559)
0619 22:28:30.632145 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:30.632150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:30.633042 (+ 892us) server_negotiation.cc:407] Connection header received
0619 22:28:30.634043 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:30.634046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:30.634101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:30.634225 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:30.635404 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.635940 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.637095 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.637322 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.637432 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.641915 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:30.641920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:30.641923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:30.641951 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:30.643861 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.644633 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.644637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.644639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.644739 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.645186 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.645190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.645192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.645418 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:30.645492 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:30.646249 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 22:28:30.646378 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:28:30.648586 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.632426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:34559 (local address 127.5.176.195:48871)
0619 22:28:30.632894 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:30.632946 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:30.633826 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:30.634388 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:30.634394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:30.634681 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:30.635263 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:30.635276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.636072 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.636080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:30.636962 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:30.636971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.637478 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.637483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:30.637719 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.638447 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:30.638469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:30.641706 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:30.644013 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:30.644020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:30.644040 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:30.644481 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:30.644856 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:30.644860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:30.644862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:30.645055 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:30.645577 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:30.645584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:30.645930 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 22:28:30.647041 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0619 22:28:30.647443 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260619 22:28:30.649899 19081 heartbeater.cc:347] Connected to a master server at 127.5.176.254:34559
I20260619 22:28:30.650192 19081 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.650813 19081 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:30.651224 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:30.640949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.5.176.254:34559)
0619 22:28:30.641255 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:30.641260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:30.641566 (+ 306us) server_negotiation.cc:407] Connection header received
0619 22:28:30.641703 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:30.641707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:30.641757 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:30.641848 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:30.643071 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.643915 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.645204 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:30.645672 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:30.645750 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:30.647338 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:30.647341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:30.647343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:30.647370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:30.648897 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.649314 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.649319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.649321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.649417 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:30.650240 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:30.650244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:30.650246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:30.650472 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:30.650699 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:30.650905 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 22:28:30.651041 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
W20260619 22:28:30.652551 19081 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:34559 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 22:28:30.652693 19081 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.652894 19081 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:30.653728 19081 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:30.653914 19081 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
W20260619 22:28:30.654644 19081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 22:28:31 dist-test-slave-bt27 krb5kdc[17619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781908111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:31.127107 18813 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:31.127439 18813 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:31.128903 18615 ts_manager.cc:194] Registered new tserver with Master: de3348ea5494400bb0c4388d098f90e6 (127.5.176.193:44015)
I20260619 22:28:31.130688 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:44089
I20260619 22:28:31.382092 18947 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:31.382624 18947 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:31.383569 18615 ts_manager.cc:194] Registered new tserver with Master: 066b24781aca4da8a4cee47ef7a62670 (127.5.176.194:46845)
I20260619 22:28:31.384385 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:47371
Jun 19 22:28:31 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:31.654904 19081 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:31.655221 19081 heartbeater.cc:511] Master 127.5.176.254:34559 requested a full tablet report, sending...
I20260619 22:28:31.656105 18615 ts_manager.cc:194] Registered new tserver with Master: 2d18a47331c14e489f6a7f29aa7b0f6c (127.5.176.195:40359)
I20260619 22:28:31.656859 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:48871
Jun 19 22:28:31 dist-test-slave-bt27 krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for HTTP/127.5.176.212@KRBTEST.COM
I20260619 22:28:31.659066 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:28:31.672564 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.660714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.5.176.254:34559)
0619 22:28:31.662693 (+ 1979us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.662698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.662940 (+ 242us) server_negotiation.cc:407] Connection header received
0619 22:28:31.663182 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.663190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.663236 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.663340 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:31.664204 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.665072 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.666564 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.666948 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.667031 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.668928 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:31.668934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:31.668937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:31.668967 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:31.670693 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:31.671194 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:31.671198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:31.671200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:31.671289 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:31.671565 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:31.671569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:31.671571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:31.671781 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:31.671920 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.672255 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.672387 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1895,"thread_start_us":75,"threads_started":1}
I20260619 22:28:31.677306 18615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:28:31.682538 18615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:28:31.720852 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.713277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.5.176.194:46845)
0619 22:28:31.713594 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.713601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.714216 (+ 615us) server_negotiation.cc:407] Connection header received
0619 22:28:31.714560 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.714567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.714808 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.714953 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:31.715538 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.716839 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.718873 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.719809 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.719899 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.720059 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.720551 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.720672 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260619 22:28:31.722131 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.712223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:46845 (local address 127.0.0.1:41242)
0619 22:28:31.714050 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.714106 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:31.714341 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:31.715188 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:31.715194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:31.715219 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:31.715383 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.715392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.717003 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.717006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.718686 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.718698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.720170 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.720173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.720297 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.720338 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 22:28:31.721811 (+ 1473us) client_negotiation.cc:240] Negotiation successful
0619 22:28:31.721967 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"thread_start_us":143,"threads_started":1}
I20260619 22:28:31.725798 18873 tablet_service.cc:1467] Processing CreateTablet for tablet a541c6bc35644b74a6ca8f1961430a1a (DEFAULT_TABLE table=test-table [id=bd81f8b6456f481f873f4054e27dbe3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:31.727109 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a541c6bc35644b74a6ca8f1961430a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:31.742445 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.713205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40359 (local address 127.0.0.1:36876)
0619 22:28:31.731046 (+ 17841us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.731065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:31.731189 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:31.733048 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:31.733054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:31.733082 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:31.733197 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.733205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.734724 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.734728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.736052 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.736060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.741824 (+ 5764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.741831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.741970 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.741998 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:28:31.742160 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 22:28:31.742262 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17745,"thread_start_us":97,"threads_started":1}
I20260619 22:28:31.743145 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.713890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.5.176.195:40359)
0619 22:28:31.728569 (+ 14679us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.728577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.731315 (+ 2738us) server_negotiation.cc:407] Connection header received
0619 22:28:31.731419 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.731424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.731666 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.732841 (+ 1175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:31.733347 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.734595 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.740494 (+ 5899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.741441 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.741564 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.741742 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.742868 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.743000 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14548,"thread_start_us":90,"threads_started":1}
I20260619 22:28:31.743774 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.713379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.5.176.193:44015)
0619 22:28:31.713770 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.713780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.721117 (+ 7337us) server_negotiation.cc:407] Connection header received
0619 22:28:31.721220 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.721229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.721507 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.721676 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:31.734562 (+ 12886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.735890 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.737456 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.738414 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.738551 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.738714 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.743504 (+ 4790us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.743630 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
I20260619 22:28:31.744146 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.712959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:44015 (local address 127.0.0.1:52146)
0619 22:28:31.720961 (+ 8002us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.720978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:31.721080 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:31.734207 (+ 13127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:31.734211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:31.734232 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:31.734325 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.734331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.736051 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.736054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.737278 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.737290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.738827 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.738831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.738937 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.738973 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:28:31.743927 (+ 4954us) client_negotiation.cc:240] Negotiation successful
0619 22:28:31.744016 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7915,"thread_start_us":113,"threads_started":1}
I20260619 22:28:31.747826 18747 tablet_service.cc:1467] Processing CreateTablet for tablet a541c6bc35644b74a6ca8f1961430a1a (DEFAULT_TABLE table=test-table [id=bd81f8b6456f481f873f4054e27dbe3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:31.749117 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a541c6bc35644b74a6ca8f1961430a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:31.750290 19015 tablet_service.cc:1467] Processing CreateTablet for tablet a541c6bc35644b74a6ca8f1961430a1a (DEFAULT_TABLE table=test-table [id=bd81f8b6456f481f873f4054e27dbe3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:31.751477 19015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a541c6bc35644b74a6ca8f1961430a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:31.762842 19117 tablet_bootstrap.cc:492] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: Bootstrap starting.
I20260619 22:28:31.765659 19117 tablet_bootstrap.cc:654] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:31.766732 19117 log.cc:824] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:31.771761 19116 tablet_bootstrap.cc:492] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Bootstrap starting.
I20260619 22:28:31.774621 19116 tablet_bootstrap.cc:654] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:31.775672 19116 log.cc:824] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:31.778975 19116 tablet_bootstrap.cc:492] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: No bootstrap required, opened a new log
I20260619 22:28:31.779224 19116 ts_tablet_manager.cc:1397] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 22:28:31.783941 19116 raft_consensus.cc:348] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.784327 19116 raft_consensus.cc:374] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:31.784410 19116 raft_consensus.cc:729] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066b24781aca4da8a4cee47ef7a62670, State: Initialized, Role: FOLLOWER
I20260619 22:28:31.785817 19116 consensus_queue.cc:260] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.787083 19116 ts_tablet_manager.cc:1428] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260619 22:28:31.787310 19117 tablet_bootstrap.cc:492] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: No bootstrap required, opened a new log
I20260619 22:28:31.787504 18947 heartbeater.cc:503] Master 127.5.176.254:34559 was elected leader, sending a full tablet report...
I20260619 22:28:31.787562 19117 ts_tablet_manager.cc:1397] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20260619 22:28:31.793504 19119 tablet_bootstrap.cc:492] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Bootstrap starting.
I20260619 22:28:31.797801 19117 raft_consensus.cc:348] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.798085 19117 raft_consensus.cc:374] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:31.798197 19117 raft_consensus.cc:729] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3348ea5494400bb0c4388d098f90e6, State: Initialized, Role: FOLLOWER
I20260619 22:28:31.798727 19117 consensus_queue.cc:260] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.800904 19119 tablet_bootstrap.cc:654] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:31.801688 19119 log.cc:824] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:31.807562 19119 tablet_bootstrap.cc:492] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: No bootstrap required, opened a new log
I20260619 22:28:31.807818 19119 ts_tablet_manager.cc:1397] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260619 22:28:31.808001 19117 ts_tablet_manager.cc:1428] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260619 22:28:31.813135 19119 raft_consensus.cc:348] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.813522 19119 raft_consensus.cc:374] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:31.813594 19119 raft_consensus.cc:729] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d18a47331c14e489f6a7f29aa7b0f6c, State: Initialized, Role: FOLLOWER
I20260619 22:28:31.814235 19119 consensus_queue.cc:260] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.815521 19119 ts_tablet_manager.cc:1428] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260619 22:28:31.815943 19081 heartbeater.cc:503] Master 127.5.176.254:34559 was elected leader, sending a full tablet report...
I20260619 22:28:31.820700 18813 heartbeater.cc:503] Master 127.5.176.254:34559 was elected leader, sending a full tablet report...
I20260619 22:28:31.824232 19121 raft_consensus.cc:482] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:31.824425 19121 raft_consensus.cc:504] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.828191 19121 leader_election.cc:288] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d18a47331c14e489f6a7f29aa7b0f6c (127.5.176.195:40359), de3348ea5494400bb0c4388d098f90e6 (127.5.176.193:44015)
I20260619 22:28:31.837827 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.826317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:33971 (local address 127.5.176.195:40359)
0619 22:28:31.826467 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.826472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.831557 (+ 5085us) server_negotiation.cc:407] Connection header received
0619 22:28:31.831752 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.831756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.831807 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.831854 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:31.832288 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.833091 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.836121 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.836749 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.836854 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.836908 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.837584 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.837679 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 22:28:31.838332 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.825932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40359 (local address 127.5.176.194:33971)
0619 22:28:31.831449 (+ 5517us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.831466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:31.831639 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:31.832006 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:31.832009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:31.832026 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:31.832171 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.832177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.833233 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.833236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.835952 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.835967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.837021 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.837024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.837139 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.837165 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 22:28:31.838062 (+ 897us) client_negotiation.cc:240] Negotiation successful
0619 22:28:31.838181 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5429,"thread_start_us":128,"threads_started":1}
I20260619 22:28:31.839135 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a541c6bc35644b74a6ca8f1961430a1a" candidate_uuid: "066b24781aca4da8a4cee47ef7a62670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" is_pre_election: true
I20260619 22:28:31.839512 19029 raft_consensus.cc:2454] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066b24781aca4da8a4cee47ef7a62670 in term 0.
I20260619 22:28:31.840055 18834 leader_election.cc:302] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066b24781aca4da8a4cee47ef7a62670, 2d18a47331c14e489f6a7f29aa7b0f6c; no voters:
I20260619 22:28:31.842342 19121 raft_consensus.cc:2781] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:28:31.842483 19121 raft_consensus.cc:482] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:28:31.842545 19121 raft_consensus.cc:3037] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:31.843290 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.828790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:32865 (local address 127.5.176.193:44015)
0619 22:28:31.828922 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.828928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.837493 (+ 8565us) server_negotiation.cc:407] Connection header received
0619 22:28:31.838929 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.838933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.838988 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.839037 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:31.839519 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.840224 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.841797 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.842512 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.842625 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.842686 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.843065 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.843160 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 22:28:31.843755 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.828418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:44015 (local address 127.5.176.194:32865)
0619 22:28:31.837388 (+ 8970us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.837404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:31.838774 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:31.839183 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:31.839185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:31.839203 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:31.839317 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.839324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.840365 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.840368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.841643 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:31.841655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.842787 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.842790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:31.842909 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.842938 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:28:31.843518 (+ 580us) client_negotiation.cc:240] Negotiation successful
0619 22:28:31.843607 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8890,"thread_start_us":117,"threads_started":1}
I20260619 22:28:31.844022 19121 raft_consensus.cc:504] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
I20260619 22:28:31.844456 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a541c6bc35644b74a6ca8f1961430a1a" candidate_uuid: "066b24781aca4da8a4cee47ef7a62670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3348ea5494400bb0c4388d098f90e6" is_pre_election: true
I20260619 22:28:31.844797 18767 raft_consensus.cc:2454] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066b24781aca4da8a4cee47ef7a62670 in term 0.
I20260619 22:28:31.844810 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a541c6bc35644b74a6ca8f1961430a1a" candidate_uuid: "066b24781aca4da8a4cee47ef7a62670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c"
I20260619 22:28:31.844941 19029 raft_consensus.cc:3037] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:31.846340 19029 raft_consensus.cc:2454] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066b24781aca4da8a4cee47ef7a62670 in term 1.
I20260619 22:28:31.846704 18834 leader_election.cc:302] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066b24781aca4da8a4cee47ef7a62670, 2d18a47331c14e489f6a7f29aa7b0f6c; no voters:
I20260619 22:28:31.846939 19122 raft_consensus.cc:2781] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:31.847198 19122 raft_consensus.cc:686] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 1 LEADER]: Becoming Leader. State: Replica: 066b24781aca4da8a4cee47ef7a62670, State: Running, Role: LEADER
I20260619 22:28:31.847597 19122 consensus_queue.cc:237] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } }
W20260619 22:28:31.847779 18814 tablet.cc:2369] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:28:31.851322 18614 catalog_manager.cc:5495] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 066b24781aca4da8a4cee47ef7a62670 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "066b24781aca4da8a4cee47ef7a62670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066b24781aca4da8a4cee47ef7a62670" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260619 22:28:31.853667 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a541c6bc35644b74a6ca8f1961430a1a" candidate_uuid: "066b24781aca4da8a4cee47ef7a62670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3348ea5494400bb0c4388d098f90e6"
I20260619 22:28:31.853818 18767 raft_consensus.cc:3037] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:31.855304 18767 raft_consensus.cc:2454] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066b24781aca4da8a4cee47ef7a62670 in term 1.
I20260619 22:28:31.856148 19121 leader_election.cc:288] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [CANDIDATE]: Term 1 election: Requested vote from peers 2d18a47331c14e489f6a7f29aa7b0f6c (127.5.176.195:40359), de3348ea5494400bb0c4388d098f90e6 (127.5.176.193:44015)
W20260619 22:28:31.857983 18948 tablet.cc:2369] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:28:31.876809 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.870796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.5.176.194:46845)
0619 22:28:31.870928 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:31.870933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:31.870959 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:28:31.871053 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:31.871056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:31.871125 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:31.871240 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:28:31.871647 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.872451 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.873732 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:31.874082 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:31.874217 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:31.874685 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:28:31.875301 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:28:31.875426 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:31.876124 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 22:28:31.876193 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 22:28:31.882692 19082 tablet.cc:2369] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:28:31.901502 18767 raft_consensus.cc:1261] T a541c6bc35644b74a6ca8f1961430a1a P de3348ea5494400bb0c4388d098f90e6 [term 1 FOLLOWER]: Refusing update from remote peer 066b24781aca4da8a4cee47ef7a62670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:31.902873 19029 raft_consensus.cc:1261] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 1 FOLLOWER]: Refusing update from remote peer 066b24781aca4da8a4cee47ef7a62670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:31.903847 19121 consensus_queue.cc:1035] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3348ea5494400bb0c4388d098f90e6" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:31.904989 19121 consensus_queue.cc:1035] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d18a47331c14e489f6a7f29aa7b0f6c" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:31.934540 19132 mvcc.cc:205] Tried to move back new op lower bound from 7298695626337652736 to 7298695626131542016. Current Snapshot: MvccSnapshot[applied={T|T < 7298695626337652736}]
I20260619 22:28:31.951679 18615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:31.951889 18615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:31.954720 18615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a4756fcf07547d9a99699269f203ba7: Sending DeleteTablet for 3 replicas of tablet a541c6bc35644b74a6ca8f1961430a1a
I20260619 22:28:31.955844 19015 tablet_service.cc:1514] Processing DeleteTablet for tablet a541c6bc35644b74a6ca8f1961430a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:31 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:36876
I20260619 22:28:31.955962 18873 tablet_service.cc:1514] Processing DeleteTablet for tablet a541c6bc35644b74a6ca8f1961430a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:31 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:41242
I20260619 22:28:31.956305 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet a541c6bc35644b74a6ca8f1961430a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:31 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52146
I20260619 22:28:31.957563 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18653
I20260619 22:28:31.966820 19140 tablet_replica.cc:330] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: stopping tablet replica
I20260619 22:28:31.967262 19140 raft_consensus.cc:2229] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:31.967672 19140 raft_consensus.cc:2258] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:31.969533 19140 ts_tablet_manager.cc:1905] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:31.970378 19141 tablet_replica.cc:330] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: stopping tablet replica
I20260619 22:28:31.970734 19141 raft_consensus.cc:2229] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:28:31.971211 19141 raft_consensus.cc:2258] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:31.972579 19140 ts_tablet_manager.cc:1918] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:31.972688 19140 log.cc:1196] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/a541c6bc35644b74a6ca8f1961430a1a
I20260619 22:28:31.972723 19141 ts_tablet_manager.cc:1905] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:31.973047 19140 ts_tablet_manager.cc:1939] T a541c6bc35644b74a6ca8f1961430a1a P 2d18a47331c14e489f6a7f29aa7b0f6c: Deleting consensus metadata
W20260619 22:28:31.974586 18602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:44015 (error 108)
I20260619 22:28:31.975473 19141 ts_tablet_manager.cc:1918] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:31.975575 19141 log.cc:1196] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/a541c6bc35644b74a6ca8f1961430a1a
I20260619 22:28:31.975545 18599 catalog_manager.cc:4841] TS 2d18a47331c14e489f6a7f29aa7b0f6c (127.5.176.195:40359): tablet a541c6bc35644b74a6ca8f1961430a1a (table test-table [id=bd81f8b6456f481f873f4054e27dbe3b]) successfully deleted
I20260619 22:28:31.975873 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18817
I20260619 22:28:31.975911 19141 ts_tablet_manager.cc:1939] T a541c6bc35644b74a6ca8f1961430a1a P 066b24781aca4da8a4cee47ef7a62670: Deleting consensus metadata
I20260619 22:28:31.978190 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:31.977766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:44015 (local address 127.0.0.1:52152)
0619 22:28:31.977970 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.978050 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:44015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 22:28:31.978716 18602 catalog_manager.cc:4584] TS de3348ea5494400bb0c4388d098f90e6 (127.5.176.193:44015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a541c6bc35644b74a6ca8f1961430a1a: Network error: Client connection negotiation failed: client connection to 127.5.176.193:44015: connect: Connection refused (error 111)
W20260619 22:28:31.984958 19110 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:28:31.984152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:46845 (local address 127.0.0.1:41260)
0619 22:28:31.984341 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:31.984357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:31.984467 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:31.984837 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:46845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 22:28:31.985569 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18951
I20260619 22:28:31.993237 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 18582
I20260619 22:28:32.002054 5827 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 22:28:32.337515 5827 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 22:28:32.337618 5827 mini_ranger.cc:66] Stopping Ranger...
I20260619 22:28:32.682981 5827 mini_ranger.cc:68] Stopped Ranger
2026-06-19 22:28:32.683 UTC [17649] LOG: received smart shutdown request
2026-06-19 22:28:32.686 UTC [17649] LOG: background worker "logical replication launcher" (PID 17658) exited with exit code 1
2026-06-19 22:28:32.687 UTC [17653] LOG: shutting down
2026-06-19 22:28:32.850 UTC [17649] LOG: database system is shut down
2026-06-19T22:28:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48302 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:28:33 dist-test-slave-bt27 krb5kdc[19152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:28:33 dist-test-slave-bt27 krb5kdc[19152](info): set up 2 sockets
Jun 19 22:28:33 dist-test-slave-bt27 krb5kdc[19152](info): commencing operation
krb5kdc: starting...
W20260619 22:28:34.026204 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:28:34.149173 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45649
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:45649,127.5.176.253:44539,127.5.176.252:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:34.264268 19165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:34.264645 19165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:34.264729 19165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:34.268927 19165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:34.269044 19165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:34.269080 19165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:34.269124 19165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:34.269157 19165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:34.274667 19165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:45649,127.5.176.253:44539,127.5.176.252:46757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:34.275851 19165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:34.276906 19165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:34.285573 19170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:34.285710 19165 server_base.cc:1034] running on GCE node
W20260619 22:28:34.285563 19173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:34.285936 19171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:34.286707 19165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:34.287891 19165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:34.289113 19165 hybrid_clock.cc:648] HybridClock initialized: now 1781908114289087 us; error 42 us; skew 500 ppm
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:34.292172 19165 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:34.293141 19165 webserver.cc:460] Webserver started at http://127.5.176.254:43821/ using document root <none> and password file <none>
I20260619 22:28:34.293706 19165 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:34.293768 19165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:34.294010 19165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:34.296218 19165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "b8b25016e3ed4579b2eb6bad2763fad7"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "3260518c5e23c8a7b69da2770748d8c7"
server_key_iv: "9a919dcb06d40b65d66094aaa390c074"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.296772 19165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "b8b25016e3ed4579b2eb6bad2763fad7"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "3260518c5e23c8a7b69da2770748d8c7"
server_key_iv: "9a919dcb06d40b65d66094aaa390c074"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.300670 19165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:28:34.303321 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:34.304654 19165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:28:34.304798 19165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "b8b25016e3ed4579b2eb6bad2763fad7"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "3260518c5e23c8a7b69da2770748d8c7"
server_key_iv: "9a919dcb06d40b65d66094aaa390c074"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.304919 19165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:34.333052 19165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:34.337455 19165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:34.337679 19165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:34.347486 19232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:45649 every 8 connection(s)
I20260619 22:28:34.347492 19165 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:45649
I20260619 22:28:34.348881 19165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:34.350704 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19165
I20260619 22:28:34.350857 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:28:34.351183 5827 external_mini_cluster.cc:1428] Setting key 184a7ba67409e28d9cb7885d2d62f2ed
I20260619 22:28:34.352113 19233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:34.356688 19233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.253@KRBTEST.COM" created.
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:34.375011 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.359932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.5.176.254:45649)
0619 22:28:34.360566 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.360580 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.360689 (+ 109us) server_negotiation.cc:407] Connection header received
0619 22:28:34.362179 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.362186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.362408 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.362571 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.363622 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.364615 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.366009 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.366558 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.366781 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.369613 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.369622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.369636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.369669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.371840 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.372345 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.372351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.372354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.372502 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.372790 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.372792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.372793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.373294 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.373393 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.373808 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.374133 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":504,"spinlock_wait_cycles":57728,"thread_start_us":173,"threads_started":1}
I20260619 22:28:34.375020 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.359932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45649 (local address 127.0.0.1:58974)
0619 22:28:34.360494 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.360560 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.361783 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.362627 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.362633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.363117 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.363429 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.363449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.364765 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.364768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.365873 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.365885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.366727 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.366730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.366957 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.367608 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.367625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.369437 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.371973 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.371981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.371991 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.372223 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.372600 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.372602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.372603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.372712 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.373396 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.373402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.373540 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.373681 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.374037 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":67,"thread_start_us":170,"threads_started":1}
I20260619 22:28:34.377349 19233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:28:34.378245 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.377886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:47960)
0619 22:28:34.378045 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.378167 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 22:28:34.378419 19181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
I20260619 22:28:34.379269 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.379091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:47976)
0619 22:28:34.379176 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.379222 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 22:28:34.379426 19233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:44539: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab.
Entry for principal kudu/127.5.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab.
I20260619 22:28:34.381042 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:44539
--webserver_interface=127.5.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab
--principal=kudu/127.5.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:45649,127.5.176.253:44539,127.5.176.252:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:34.428690 19233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } attempt: 1
I20260619 22:28:34.429625 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.429266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:47980)
0619 22:28:34.429453 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.429532 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 22:28:34.430521 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.430254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:47986)
0619 22:28:34.430391 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.430448 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:28:34.430721 19233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:44539: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
W20260619 22:28:34.492816 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:34.493268 19240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:34.493417 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:34.497638 19240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:34.497714 19240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:34.497735 19240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:34.497752 19240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:34.497767 19240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:34.498939 19233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } attempt: 2
I20260619 22:28:34.499833 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.499488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:47992)
0619 22:28:34.499662 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.499735 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 22:28:34.500640 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.500406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:47998)
0619 22:28:34.500522 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.500576 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 22:28:34.500797 19233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:44539: Network error: Client connection negotiation failed: client connection to 127.5.176.253:44539: connect: Connection refused (error 111)
I20260619 22:28:34.503945 19240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:45649,127.5.176.253:44539,127.5.176.252:46757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab
--principal=kudu/127.5.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:34.505125 19240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:34.506234 19240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:28:34.514918 19240 server_base.cc:1034] running on GCE node
W20260619 22:28:34.514871 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:34.514858 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:34.514873 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:34.515692 19240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:34.516868 19240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:34.518072 19240 hybrid_clock.cc:648] HybridClock initialized: now 1781908114518041 us; error 45 us; skew 500 ppm
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:34.520913 19240 init.cc:377] Logged in from keytab as kudu/127.5.176.253@KRBTEST.COM (short username kudu)
I20260619 22:28:34.521852 19240 webserver.cc:460] Webserver started at http://127.5.176.253:39113/ using document root <none> and password file <none>
I20260619 22:28:34.522475 19240 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:34.522564 19240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:34.522794 19240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:34.524813 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data/instance:
uuid: "05bfee27812e46f9bef8d3f6547b838c"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "00a593e8a4005d1355dd774dd27df08c"
server_key_iv: "5e959ad8c614b20ff7fe6fff64236835"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.525378 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal/instance:
uuid: "05bfee27812e46f9bef8d3f6547b838c"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "00a593e8a4005d1355dd774dd27df08c"
server_key_iv: "5e959ad8c614b20ff7fe6fff64236835"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.529038 19240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260619 22:28:34.531507 19255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:34.532658 19240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:28:34.532805 19240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal
uuid: "05bfee27812e46f9bef8d3f6547b838c"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "00a593e8a4005d1355dd774dd27df08c"
server_key_iv: "5e959ad8c614b20ff7fe6fff64236835"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.532920 19240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:34.547833 19240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:34.551586 19240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:34.551803 19240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:34.559298 19240 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.253:44539
I20260619 22:28:34.559321 19307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.253:44539 every 8 connection(s)
I20260619 22:28:34.560319 19240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
I20260619 22:28:34.563306 19308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:34.566879 19308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:28:34.567509 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19240
I20260619 22:28:34.567646 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-1/wal/instance
I20260619 22:28:34.567929 5827 external_mini_cluster.cc:1428] Setting key 2a8fb9c28e2a77397ff75d67f857daa6
WARNING: no policy specified for kudu/127.5.176.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.252@KRBTEST.COM" created.
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:34.583288 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.569016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.5.176.254:45649)
0619 22:28:34.569179 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.569185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.570043 (+ 858us) server_negotiation.cc:407] Connection header received
0619 22:28:34.570995 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.570999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.571061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.571161 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.572306 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.572843 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.573991 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.574292 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.574415 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.578793 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.578797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.578799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.578826 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.580542 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.581238 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.581241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.581242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.581325 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.581780 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.581784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.581785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.582118 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.582281 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.583006 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.583141 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:28:34.584631 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.569420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45649 (local address 127.0.0.1:58982)
0619 22:28:34.569870 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.569931 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.570772 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.571327 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.571334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.571674 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.572149 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.572161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.572972 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.572976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.573835 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.573844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.574445 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.574450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.574634 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.575273 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.575300 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.578598 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.580672 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.580681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.580702 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.581100 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.581440 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.581446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.581451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.581649 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.582280 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.582288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.582685 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.582928 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.583347 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260619 22:28:34.586309 19308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
I20260619 22:28:34.591977 19233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab.
Entry for principal kudu/127.5.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab.
I20260619 22:28:34.595714 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:46757
--webserver_interface=127.5.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab
--principal=kudu/127.5.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:45649,127.5.176.253:44539,127.5.176.252:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:34.597223 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.586822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:48006)
0619 22:28:34.586966 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.586982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.587061 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.587646 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.587649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.587832 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.587915 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.587921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.588980 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.588983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.589584 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.589599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.590290 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.590293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.590383 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.590939 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.590974 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.592417 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.595299 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.595306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.595310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.595529 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.595894 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.595897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.595899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.595985 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.596542 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.596545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.596638 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.596792 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.596962 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
I20260619 22:28:34.598491 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.586966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.5.176.253:44539)
0619 22:28:34.587278 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.587287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.587304 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:28:34.587336 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.587340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.587527 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.587670 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.588052 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.588849 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.589697 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.590122 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.590290 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.592797 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.592810 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.592859 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.592886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.595171 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.595660 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.595666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.595668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.595759 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.596077 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.596080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.596082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.596417 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.596585 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.596949 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.597911 (+ 962us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
I20260619 22:28:34.600648 19308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:28:34.601449 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.601173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39540)
0619 22:28:34.601304 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.601395 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 22:28:34.601699 19259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
I20260619 22:28:34.602751 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.602522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39552)
0619 22:28:34.602648 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.602696 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 22:28:34.602809 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.592428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:48020)
0619 22:28:34.592620 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.592641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.592758 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.593191 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.593194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.593364 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.593920 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.593930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.594904 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.594908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.595709 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.595720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.596340 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.596347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.596497 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.597007 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.597025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.598605 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.601063 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.601070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.601075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.601404 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.601762 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.601766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.601768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.601861 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.602332 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.602336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.602438 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.602505 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.602650 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 22:28:34.602939 19308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:46757: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
I20260619 22:28:34.603404 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.592524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.5.176.253:44539)
0619 22:28:34.592888 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.592896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.592908 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:28:34.592936 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.592938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.592997 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.593128 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.594073 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.594740 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.595834 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.596207 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.596319 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.598768 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.598772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.598774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.598797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.600391 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.601570 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.601576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.601580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.601663 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.601970 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.601975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.601977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.602216 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.603144 (+ 928us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.603204 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.603298 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260619 22:28:34.604121 19233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:28:34.604910 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.604578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39554)
0619 22:28:34.604771 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.604831 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 22:28:34.605675 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.605432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39568)
0619 22:28:34.605578 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.605630 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 22:28:34.605822 19233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:46757: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
I20260619 22:28:34.636993 19233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } attempt: 1
I20260619 22:28:34.637862 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.637554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39572)
0619 22:28:34.637709 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.637780 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 22:28:34.638727 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.638492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39574)
0619 22:28:34.638611 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.638661 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 22:28:34.638909 19233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:46757: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
I20260619 22:28:34.652235 19308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } attempt: 1
I20260619 22:28:34.653167 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.652835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39576)
0619 22:28:34.653012 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.653079 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 22:28:34.653887 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.653666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39582)
0619 22:28:34.653779 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.653833 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:28:34.654031 19308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:46757: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
I20260619 22:28:34.714087 19233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } attempt: 2
I20260619 22:28:34.715060 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.714732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39594)
0619 22:28:34.714910 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.714973 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 22:28:34.714998 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:34.715374 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:34.715457 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 22:28:34.715865 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.715652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39596)
0619 22:28:34.715761 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.715814 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:28:34.716020 19233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:46757: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
W20260619 22:28:34.719794 19315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:34.719862 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:34.719887 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:34.719934 19315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:34.719980 19315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:34.722215 19308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } attempt: 2
I20260619 22:28:34.722975 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.722691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39598)
0619 22:28:34.722828 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.722897 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:28:34.723673 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.723467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39608)
0619 22:28:34.723577 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.723625 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:28:34.723802 19308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:46757: Network error: Client connection negotiation failed: client connection to 127.5.176.252:46757: connect: Connection refused (error 111)
I20260619 22:28:34.725803 19315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:45649,127.5.176.253:44539,127.5.176.252:46757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab
--principal=kudu/127.5.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:34.727044 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:34.728109 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:34.736604 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:34.736606 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:34.736678 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:34.737021 19315 server_base.cc:1034] running on GCE node
I20260619 22:28:34.737638 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:34.738852 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:34.740065 19315 hybrid_clock.cc:648] HybridClock initialized: now 1781908114740032 us; error 46 us; skew 500 ppm
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:34.742992 19315 init.cc:377] Logged in from keytab as kudu/127.5.176.252@KRBTEST.COM (short username kudu)
I20260619 22:28:34.743950 19315 webserver.cc:460] Webserver started at http://127.5.176.252:36951/ using document root <none> and password file <none>
I20260619 22:28:34.744517 19315 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:34.744621 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:34.744859 19315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:34.747020 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data/instance:
uuid: "c5035bf64d2043538894012429927092"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "6d078b7c24ec6b2621c7ac74bcbc3ae4"
server_key_iv: "3e1f166d1ff2f4100d1432d0ecd365b1"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.747577 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal/instance:
uuid: "c5035bf64d2043538894012429927092"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "6d078b7c24ec6b2621c7ac74bcbc3ae4"
server_key_iv: "3e1f166d1ff2f4100d1432d0ecd365b1"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.751426 19315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:28:34.754706 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:34.756294 19315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:28:34.756431 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal
uuid: "c5035bf64d2043538894012429927092"
format_stamp: "Formatted at 2026-06-19 22:28:34 on dist-test-slave-bt27"
server_key: "6d078b7c24ec6b2621c7ac74bcbc3ae4"
server_key_iv: "3e1f166d1ff2f4100d1432d0ecd365b1"
server_key_version: "encryptionkey@0"
I20260619 22:28:34.756556 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:34.767174 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:34.770083 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:34.770352 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:34.777456 19315 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.252:46757
I20260619 22:28:34.777503 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.252:46757 every 8 connection(s)
I20260619 22:28:34.778535 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
I20260619 22:28:34.781306 19384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:34.782922 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19315
I20260619 22:28:34.783067 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781907951519384-5827-0/minicluster-data/master-2/wal/instance
I20260619 22:28:34.783329 5827 external_mini_cluster.cc:1428] Setting key 472da1560ec6410c0bed865e969610ce
I20260619 22:28:34.785508 19384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:34.798027 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.787796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.5.176.254:45649)
0619 22:28:34.787984 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.787989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.788286 (+ 297us) server_negotiation.cc:407] Connection header received
0619 22:28:34.788407 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.788410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.788464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.788534 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.789282 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.790024 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.791262 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.791569 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.791701 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.794366 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.794369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.794371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.794395 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.796010 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.796585 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.796589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.796591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.796679 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.796974 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.796977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.796978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.797168 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.797252 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.797711 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.797830 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:28:34.799166 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.787740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45649 (local address 127.0.0.1:58992)
0619 22:28:34.788155 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.788202 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.788315 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.788601 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.788606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.789027 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.789172 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.789177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.790183 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.790185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.791137 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.791145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.791764 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.791768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.791862 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.792376 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.792392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.794219 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.796140 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.796148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.796156 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.796485 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.796774 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.796777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.796779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.796852 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.797288 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.797296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.797429 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.797626 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.797981 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":111,"thread_start_us":157,"threads_started":1}
I20260619 22:28:34.799166 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.784707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.5.176.252:46757)
0619 22:28:34.785147 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.785162 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.785197 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:28:34.785827 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.785843 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.786264 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.786554 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.787504 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.788951 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.790173 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.790931 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.791088 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.793758 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.793763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.793773 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.793824 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.795714 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.796243 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.796250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.796252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.796374 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.796701 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.796703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.796704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.797208 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.797415 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.797638 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.797935 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":300,"thread_start_us":175,"threads_started":1}
I20260619 22:28:34.800516 19384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:34.810496 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.801023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:44539 (local address 127.0.0.1:48034)
0619 22:28:34.801259 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.801274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.801361 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.801768 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.801774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.801972 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.802134 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.802144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.803033 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.803036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.803843 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.803853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.804356 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.804359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.804429 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.804935 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.804952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.806477 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.808768 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.808771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.808773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.809041 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.809368 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.809371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.809373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.809466 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.809893 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.809896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.810004 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.810119 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.810288 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260619 22:28:34.810496 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.801145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.5.176.253:44539)
0619 22:28:34.801258 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.801263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.801318 (+ 55us) server_negotiation.cc:407] Connection header received
0619 22:28:34.801473 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.801476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.801552 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.801632 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.802277 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.802893 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.803969 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.804242 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.804350 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.806663 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.806667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.806669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.806695 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.808582 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.809152 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.809157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.809159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.809254 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.809573 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.809577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.809579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.809778 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.809905 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.810210 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.810337 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 22:28:34.810626 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.801700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.5.176.254:45649)
0619 22:28:34.801800 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.801803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.801815 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:28:34.801929 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.801931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.801967 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.802031 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.802727 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.803290 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.804244 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.804550 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.804677 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.807245 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.807248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.807251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.807276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.808779 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.809220 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.809224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.809226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.809338 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.809632 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.809636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.809637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.809822 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.810068 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.810249 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.810366 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 22:28:34.812131 19384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:28:34.814965 19308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } attempt: 3
I20260619 22:28:34.815199 19233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } attempt: 3
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
Jun 19 22:28:34 dist-test-slave-bt27 krb5kdc[19152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
I20260619 22:28:34.822341 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.812749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.5.176.252:46757)
0619 22:28:34.812871 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.812876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.812895 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:28:34.812965 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.812969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.813025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.813121 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.813569 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.814062 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.814842 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.815212 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.815342 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.818586 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.818589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.818591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.818613 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.820439 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.820935 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.820939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.820941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.821017 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.821310 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.821316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.821321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.821550 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.821622 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.822005 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.822130 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:28:34.823247 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.812663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39628)
0619 22:28:34.812783 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.812795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.812865 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.813199 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.813203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.813368 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.813454 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.813461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.814204 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.814210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.814704 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.814712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.815326 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.815330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.815409 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.816797 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.816810 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.818466 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.820554 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.820559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.820561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.820779 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.821128 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.821134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.821138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.821219 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.821684 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.821690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.821836 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.822835 (+ 999us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.823054 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 22:28:34.823964 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.811871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.5.176.253:44539)
0619 22:28:34.812066 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.812073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.812087 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:28:34.812124 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.812126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.812174 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.812263 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.812981 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.813772 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.814618 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.814842 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.814949 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.818705 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.818709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.818712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.818740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.821650 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.822226 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.822230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.822231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.822305 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.822651 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.822657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.822661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.822905 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.823142 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.823604 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.823741 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 22:28:34.825464 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.815353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39636)
0619 22:28:34.815605 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.815618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.815704 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.816262 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.816266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.816476 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.816564 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.816571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.817688 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.817695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.818426 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.818437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.819078 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.819081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.819165 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.819634 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.819645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.821275 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.823988 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.823991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.823993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.824234 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.824551 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.824554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.824556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.824663 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.825093 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.825099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.825199 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.825263 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.825363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260619 22:28:34.825950 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.815612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.5.176.252:46757)
0619 22:28:34.815991 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.816004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.816019 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:28:34.816058 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.816064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.816114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.816675 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.816797 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.817579 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.818561 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.818943 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.819062 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.821790 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.821796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.821799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.821817 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.823860 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.824355 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.824360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.824364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.824449 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.824756 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.824758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.824759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.824982 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.825644 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.825715 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.825822 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"spinlock_wait_cycles":1408,"thread_start_us":141,"threads_started":1}
I20260619 22:28:34.826880 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.816178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:46757 (local address 127.0.0.1:39646)
0619 22:28:34.816377 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:34.816393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:34.816473 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:34.816895 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:34.816901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:34.817094 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:34.817229 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.817237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.818536 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.818540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.819404 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:34.819420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.820172 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.820178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:34.820304 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.820732 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:34.820751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:34.822405 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:34.825035 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.825039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.825041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.825323 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.825690 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:34.825693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:34.825695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:34.825794 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:34.826317 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:34.826324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:34.826446 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 22:28:34.826582 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 22:28:34.826728 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 22:28:34.826896 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.816207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.5.176.252:46757)
0619 22:28:34.816640 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.816647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.816657 (+ 10us) server_negotiation.cc:407] Connection header received
0619 22:28:34.816684 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.816686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.816717 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.817368 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.817512 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.818384 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.819574 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.820010 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.820546 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.822572 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.822579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.822582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.822614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.824893 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.825471 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.825475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.825479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.825572 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.825930 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.825933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.825935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.826199 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.826311 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.826634 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.826734 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":84,"threads_started":1}
I20260619 22:28:34.827736 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: Bootstrap starting.
I20260619 22:28:34.829293 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: Bootstrap starting.
I20260619 22:28:34.830793 19384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:34.831763 19384 log.cc:824] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:34.834044 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:28:34.834254 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: No bootstrap required, opened a new log
I20260619 22:28:34.834455 19233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7: Bootstrap starting.
I20260619 22:28:34.837738 19233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:34.838879 19233 log.cc:824] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:34.839557 19384 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.839826 19384 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:34.839890 19384 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5035bf64d2043538894012429927092, State: Initialized, Role: FOLLOWER
I20260619 22:28:34.840045 19308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:34.840430 19384 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.840977 19308 log.cc:824] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:34.841902 19384 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:34.846737 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.836323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.5.176.253:44539)
0619 22:28:34.836484 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.836489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.837243 (+ 754us) server_negotiation.cc:407] Connection header received
0619 22:28:34.837399 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.837403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.837454 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.837528 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.838471 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.839253 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.840583 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.840908 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.840985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.842864 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.842868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.842872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.842902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.844665 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.845191 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.845196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.845200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.845288 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.845644 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.845651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.845654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.845906 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.846019 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.846427 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.846557 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 22:28:34.848062 19412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:28:34.848153 19412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:28:34.849095 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.836750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.5.176.252:46757)
0619 22:28:34.836848 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.836852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.838474 (+ 1622us) server_negotiation.cc:407] Connection header received
0619 22:28:34.838634 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.838637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.838688 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.838739 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.839726 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.840488 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.841774 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.842194 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.842294 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.844344 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.844348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.844351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.844382 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.846120 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.846626 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.846630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.846632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.846709 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.847094 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.847097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.847099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.847304 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.847446 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.848784 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.848922 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 22:28:34.850346 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: No bootstrap required, opened a new log
I20260619 22:28:34.853722 19308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.854019 19308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:34.854079 19308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bfee27812e46f9bef8d3f6547b838c, State: Initialized, Role: FOLLOWER
I20260619 22:28:34.846905 19315 master_runner.cc:186] Error getting master registration for 127.5.176.254:45649: OK, instance_id { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" instance_seqno: 1781908114344803 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:28:34.854521 19315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:28:34.855069 19308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.856598 19233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7: No bootstrap required, opened a new log
I20260619 22:28:34.856578 19308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:34.857669 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:34.843179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.5.176.254:45649)
0619 22:28:34.843308 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:34.843315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:34.847601 (+ 4286us) server_negotiation.cc:407] Connection header received
0619 22:28:34.847804 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:34.847808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:34.847854 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:34.847918 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:34.848828 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.849557 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.851146 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:34.851459 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:34.851535 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:34.853375 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:34.853379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:34.853381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:34.853411 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:34.855369 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.856048 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.856053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.856055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.856152 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:34.856508 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:34.856512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:34.856516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:34.856798 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:34.856929 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:34.857374 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 22:28:34.857502 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:28:34.861340 19233 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.861598 19233 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:34.861670 19233 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b25016e3ed4579b2eb6bad2763fad7, State: Initialized, Role: FOLLOWER
I20260619 22:28:34.862238 19233 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.862540 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.862798 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:28:34.863756 19233 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:34.860767 19240 master_runner.cc:186] Error getting master registration for 127.5.176.254:45649: OK, instance_id { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" instance_seqno: 1781908114344803 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:28:34.864989 19240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:28:34.865276 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.865432 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:28:34.866354 19399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.866513 19399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:28:34.868049 19425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:28:34.868166 19425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 22:28:34.869570 19437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:28:34.869648 19437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:28:34.874107 19165 master_runner.cc:263] No leader master found from master b8b25016e3ed4579b2eb6bad2763fad7
I20260619 22:28:34.874231 19165 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:28:34.882613 19426 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:34.882828 19426 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.884202 19426 leader_election.cc:288] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bfee27812e46f9bef8d3f6547b838c (127.5.176.253:44539), c5035bf64d2043538894012429927092 (127.5.176.252:46757)
I20260619 22:28:34.885401 19282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bfee27812e46f9bef8d3f6547b838c" is_pre_election: true
I20260619 22:28:34.885735 19282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8b25016e3ed4579b2eb6bad2763fad7 in term 0.
I20260619 22:28:34.886711 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5035bf64d2043538894012429927092" is_pre_election: true
I20260619 22:28:34.887061 19358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8b25016e3ed4579b2eb6bad2763fad7 in term 0.
I20260619 22:28:34.887539 19184 leader_election.cc:302] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8b25016e3ed4579b2eb6bad2763fad7, c5035bf64d2043538894012429927092; no voters:
I20260619 22:28:34.888566 19426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:28:34.888679 19426 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:28:34.888736 19426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:34.890283 19426 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.891175 19282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bfee27812e46f9bef8d3f6547b838c"
I20260619 22:28:34.891341 19282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:34.892910 19282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b25016e3ed4579b2eb6bad2763fad7 in term 1.
I20260619 22:28:34.893347 19181 leader_election.cc:302] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bfee27812e46f9bef8d3f6547b838c, b8b25016e3ed4579b2eb6bad2763fad7; no voters:
I20260619 22:28:34.897955 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5035bf64d2043538894012429927092"
I20260619 22:28:34.898176 19358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:34.899722 19358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b25016e3ed4579b2eb6bad2763fad7 in term 1.
I20260619 22:28:34.900130 19426 leader_election.cc:288] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [CANDIDATE]: Term 1 election: Requested vote from peers 05bfee27812e46f9bef8d3f6547b838c (127.5.176.253:44539), c5035bf64d2043538894012429927092 (127.5.176.252:46757)
I20260619 22:28:34.900334 19426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:34.900444 19426 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [term 1 LEADER]: Becoming Leader. State: Replica: b8b25016e3ed4579b2eb6bad2763fad7, State: Running, Role: LEADER
I20260619 22:28:34.900827 19426 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } }
I20260619 22:28:34.902858 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b25016e3ed4579b2eb6bad2763fad7. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.902989 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:34.910497 19442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:34.911938 19442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:34.921110 19358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [term 1 FOLLOWER]: Refusing update from remote peer b8b25016e3ed4579b2eb6bad2763fad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:34.921927 19440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:34.926544 19399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b25016e3ed4579b2eb6bad2763fad7. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.926726 19399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:28:34.928077 19282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [term 1 FOLLOWER]: Refusing update from remote peer b8b25016e3ed4579b2eb6bad2763fad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:34.928858 19440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:34.937693 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b25016e3ed4579b2eb6bad2763fad7. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.937911 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:28:34.947851 19440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.948047 19440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:34.952468 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.952641 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:28:34.953135 19443 mvcc.cc:205] Tried to move back new op lower bound from 7298695638695993344 to 7298695638637338624. Current Snapshot: MvccSnapshot[applied={T|T < 7298695638695993344}]
I20260619 22:28:34.954340 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.954488 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:34.955848 19399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b25016e3ed4579b2eb6bad2763fad7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45649 } } peers { permanent_uuid: "05bfee27812e46f9bef8d3f6547b838c" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 44539 } } peers { permanent_uuid: "c5035bf64d2043538894012429927092" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 46757 } } }
I20260619 22:28:34.955994 19399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:28:34.956897 19446 mvcc.cc:205] Tried to move back new op lower bound from 7298695638695993344 to 7298695638637338624. Current Snapshot: MvccSnapshot[applied={T|T < 7298695638695993344}]
I20260619 22:28:34.957110 19444 mvcc.cc:205] Tried to move back new op lower bound from 7298695638695993344 to 7298695638637338624. Current Snapshot: MvccSnapshot[applied={T|T < 7298695638695993344}]
I20260619 22:28:34.965451 19442 catalog_manager.cc:1347] Generated new cluster ID: 2aa4c6951c1c43379aba7249defaaf6b
I20260619 22:28:34.965569 19442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:34.988737 19442 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:28:34.990000 19442 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:35.006841 19442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b25016e3ed4579b2eb6bad2763fad7: Generated new TSK 0
I20260619 22:28:35.007747 19442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:28:35.849447 19412 catalog_manager.cc:1259] Loaded cluster ID: 2aa4c6951c1c43379aba7249defaaf6b
I20260619 22:28:35.849530 19412 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: loading cluster ID for follower catalog manager: success
I20260619 22:28:35.851266 19412 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: acquiring CA information for follower catalog manager: success
I20260619 22:28:35.852576 19412 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5035bf64d2043538894012429927092: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:28:35.869586 19425 catalog_manager.cc:1259] Loaded cluster ID: 2aa4c6951c1c43379aba7249defaaf6b
I20260619 22:28:35.869678 19425 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: loading cluster ID for follower catalog manager: success
I20260619 22:28:35.871445 19425 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: acquiring CA information for follower catalog manager: success
I20260619 22:28:35.872666 19425 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05bfee27812e46f9bef8d3f6547b838c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:28:36.075655 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19165
I20260619 22:28:36.082244 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19240
I20260619 22:28:36.088165 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19315
2026-06-19T22:28:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3116 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:28:36 dist-test-slave-bt27 krb5kdc[19458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:28:36 dist-test-slave-bt27 krb5kdc[19458](info): set up 2 sockets
Jun 19 22:28:36 dist-test-slave-bt27 krb5kdc[19458](info): commencing operation
krb5kdc: starting...
W20260619 22:28:37.150507 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:28:37 dist-test-slave-bt27 krb5kdc[19458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:28:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:28:37.276564 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:44183
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:44183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:37.387149 19471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:37.387430 19471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:37.387472 19471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:37.392584 19471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:37.392660 19471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:37.392681 19471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:37.392699 19471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:37.392715 19471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:37.398458 19471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:44183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:44183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:37.399536 19471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:37.400620 19471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:28:37.409312 19471 server_base.cc:1034] running on GCE node
W20260619 22:28:37.409190 19479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:37.409205 19476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:37.409192 19477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:37.410064 19471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:37.411257 19471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:37.412472 19471 hybrid_clock.cc:648] HybridClock initialized: now 1781908117412447 us; error 30 us; skew 500 ppm
Jun 19 22:28:37 dist-test-slave-bt27 krb5kdc[19458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:37.415463 19471 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:37.416414 19471 webserver.cc:460] Webserver started at http://127.5.176.254:37095/ using document root <none> and password file <none>
I20260619 22:28:37.416993 19471 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:37.417084 19471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:37.417320 19471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:37.419528 19471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "8ff4817264e540f29e51edeea4f9e8e7"
format_stamp: "Formatted at 2026-06-19 22:28:37 on dist-test-slave-bt27"
server_key: "5d842f0d4f10c70650682b180305778b"
server_key_iv: "d4b49a6765a645ef330fd38c57dda0bc"
server_key_version: "encryptionkey@0"
I20260619 22:28:37.420115 19471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "8ff4817264e540f29e51edeea4f9e8e7"
format_stamp: "Formatted at 2026-06-19 22:28:37 on dist-test-slave-bt27"
server_key: "5d842f0d4f10c70650682b180305778b"
server_key_iv: "d4b49a6765a645ef330fd38c57dda0bc"
server_key_version: "encryptionkey@0"
I20260619 22:28:37.424033 19471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 22:28:37.426559 19486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:37.427889 19471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:28:37.428035 19471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "8ff4817264e540f29e51edeea4f9e8e7"
format_stamp: "Formatted at 2026-06-19 22:28:37 on dist-test-slave-bt27"
server_key: "5d842f0d4f10c70650682b180305778b"
server_key_iv: "d4b49a6765a645ef330fd38c57dda0bc"
server_key_version: "encryptionkey@0"
I20260619 22:28:37.428153 19471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:37.443531 19471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:37.447919 19471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:37.448158 19471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:37.456689 19471 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:44183
I20260619 22:28:37.456740 19538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:44183 every 8 connection(s)
I20260619 22:28:37.457679 19471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:37.460474 19539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:37.463040 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19471
I20260619 22:28:37.463186 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:28:37.463460 5827 external_mini_cluster.cc:1428] Setting key 77ae0527653aed2c7a420132292f5da1
I20260619 22:28:37.466990 19539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7: Bootstrap starting.
I20260619 22:28:37.469633 19539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:37.470522 19539 log.cc:824] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:37.472506 19539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7: No bootstrap required, opened a new log
Jun 19 22:28:37 dist-test-slave-bt27 krb5kdc[19458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:37.476786 19539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff4817264e540f29e51edeea4f9e8e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 44183 } }
I20260619 22:28:37.477108 19539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:37.477183 19539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff4817264e540f29e51edeea4f9e8e7, State: Initialized, Role: FOLLOWER
I20260619 22:28:37.477679 19539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff4817264e540f29e51edeea4f9e8e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 44183 } }
I20260619 22:28:37.477842 19539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:37.477931 19539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:37.478050 19539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:37.479233 19539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff4817264e540f29e51edeea4f9e8e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 44183 } }
I20260619 22:28:37.479596 19539 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff4817264e540f29e51edeea4f9e8e7; no voters:
I20260619 22:28:37.479684 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:37.464891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.5.176.254:44183)
0619 22:28:37.465318 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:37.465341 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:37.465383 (+ 42us) server_negotiation.cc:407] Connection header received
0619 22:28:37.466245 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:37.466269 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:37.466691 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:37.467046 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:37.467809 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:37.469308 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:37.470277 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:37.470991 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:37.471112 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:37.473692 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:37.473699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:37.473716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:37.473745 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:37.476445 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:37.476886 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:37.476891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:37.476897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:37.477036 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:37.477335 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:37.477338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:37.477340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:37.477853 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:37.478083 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:37.478376 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 22:28:37.478684 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260619 22:28:37.479969 19539 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:28:37.480082 19544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:37.480340 19544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff4817264e540f29e51edeea4f9e8e7, State: Running, Role: LEADER
I20260619 22:28:37.480669 19544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff4817264e540f29e51edeea4f9e8e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 44183 } }
I20260619 22:28:37.481273 19539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:37.482508 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff4817264e540f29e51edeea4f9e8e7. Latest consensus state: current_term: 1 leader_uuid: "8ff4817264e540f29e51edeea4f9e8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff4817264e540f29e51edeea4f9e8e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 44183 } } }
I20260619 22:28:37.482632 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:37.482937 19545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff4817264e540f29e51edeea4f9e8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff4817264e540f29e51edeea4f9e8e7" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 44183 } } }
I20260619 22:28:37.483012 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:28:37.483024 19545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:37.483131 19553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:37.486271 19553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:37.492090 19553 catalog_manager.cc:1347] Generated new cluster ID: b251619264ca4b94bf3fdd6682714aba
I20260619 22:28:37.492200 19553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:37.500145 19553 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:28:37.501430 19553 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:37.514887 19553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff4817264e540f29e51edeea4f9e8e7: Generated new TSK 0
I20260619 22:28:37.515820 19553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:28:37.538911 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19471
2026-06-19T22:28:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1448 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:28:37 dist-test-slave-bt27 krb5kdc[19566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:28:37 dist-test-slave-bt27 krb5kdc[19566](info): set up 2 sockets
Jun 19 22:28:37 dist-test-slave-bt27 krb5kdc[19566](info): commencing operation
krb5kdc: starting...
W20260619 22:28:38.586545 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:28:38 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:28:38.709228 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45071
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:38.821934 19579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:38.822371 19579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:38.822471 19579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:38.826695 19579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:38.826808 19579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:38.826848 19579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:38.826898 19579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:38.826930 19579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:38.832927 19579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:38.834204 19579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:38.835289 19579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:38.844342 19587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:38.844364 19585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:38.844364 19584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:38.844703 19579 server_base.cc:1034] running on GCE node
I20260619 22:28:38.845342 19579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:38.846779 19579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:38.847997 19579 hybrid_clock.cc:648] HybridClock initialized: now 1781908118847963 us; error 49 us; skew 500 ppm
Jun 19 22:28:38 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908118, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:38.856585 19579 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:38.857645 19579 webserver.cc:460] Webserver started at http://127.5.176.254:37247/ using document root <none> and password file <none>
I20260619 22:28:38.858335 19579 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:38.858412 19579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:38.858606 19579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:38.860925 19579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "9698070a3daa4366b1a852dae52beac6"
format_stamp: "Formatted at 2026-06-19 22:28:38 on dist-test-slave-bt27"
server_key: "2080382f48537c94e1c6eeacb0520951"
server_key_iv: "541fd4f4e1514d9f4cdbe3cd1ac51acc"
server_key_version: "encryptionkey@0"
I20260619 22:28:38.861518 19579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "9698070a3daa4366b1a852dae52beac6"
format_stamp: "Formatted at 2026-06-19 22:28:38 on dist-test-slave-bt27"
server_key: "2080382f48537c94e1c6eeacb0520951"
server_key_iv: "541fd4f4e1514d9f4cdbe3cd1ac51acc"
server_key_version: "encryptionkey@0"
I20260619 22:28:38.865758 19579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:28:38.868505 19594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:38.870025 19579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:28:38.870222 19579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "9698070a3daa4366b1a852dae52beac6"
format_stamp: "Formatted at 2026-06-19 22:28:38 on dist-test-slave-bt27"
server_key: "2080382f48537c94e1c6eeacb0520951"
server_key_iv: "541fd4f4e1514d9f4cdbe3cd1ac51acc"
server_key_version: "encryptionkey@0"
I20260619 22:28:38.870342 19579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:38.884825 19579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:38.888125 19579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:38.888346 19579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:38.895061 19579 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:45071
I20260619 22:28:38.895385 19646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:45071 every 8 connection(s)
I20260619 22:28:38.896114 19579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:38.898846 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:38.904793 19647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap starting.
I20260619 22:28:38.906200 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19579
I20260619 22:28:38.906348 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:28:38.906626 5827 external_mini_cluster.cc:1428] Setting key 0aaa1205627956becbecc4869a78237b
I20260619 22:28:38.907481 19647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:38.908335 19647 log.cc:824] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:38.910257 19647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: No bootstrap required, opened a new log
I20260619 22:28:38.913579 19647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:38.913794 19647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:38.913858 19647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9698070a3daa4366b1a852dae52beac6, State: Initialized, Role: FOLLOWER
I20260619 22:28:38.914350 19647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:38.914475 19647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:38.914563 19647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:38.914675 19647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 22:28:38 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:38.915652 19647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:38.915990 19647 leader_election.cc:302] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9698070a3daa4366b1a852dae52beac6; no voters:
I20260619 22:28:38.916333 19647 leader_election.cc:288] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:28:38.916472 19652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:38.916659 19652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 LEADER]: Becoming Leader. State: Replica: 9698070a3daa4366b1a852dae52beac6, State: Running, Role: LEADER
I20260619 22:28:38.916949 19652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:38.917616 19647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:38.918109 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9698070a3daa4366b1a852dae52beac6. Latest consensus state: current_term: 1 leader_uuid: "9698070a3daa4366b1a852dae52beac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } } }
I20260619 22:28:38.918251 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:38.920897 19657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:38.920933 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:38.907785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.5.176.254:45071)
0619 22:28:38.908229 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:38.908244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:38.908280 (+ 36us) server_negotiation.cc:407] Connection header received
0619 22:28:38.909052 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:38.909070 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:38.909394 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:38.909737 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:38.910597 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:38.911574 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:38.912435 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:38.912917 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:38.913096 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:38.915798 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:38.915805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:38.915824 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:38.915863 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:38.917834 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:38.918350 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:38.918359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:38.918367 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:38.918512 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:38.918813 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:38.918818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:38.918821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:38.919318 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:38.919520 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:38.919792 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 22:28:38.920105 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":142,"threads_started":1}
I20260619 22:28:38.921304 19654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9698070a3daa4366b1a852dae52beac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } } }
I20260619 22:28:38.921469 19654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:38.924257 19657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:38.929695 19657 catalog_manager.cc:1347] Generated new cluster ID: 8679da619e51439aa43fec2d2e2438ad
I20260619 22:28:38.929775 19657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:38.953173 19657 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:28:38.954277 19657 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:38.963253 19657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Generated new TSK 0
I20260619 22:28:38.964109 19657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:28:39.000783 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:45071
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:39.115808 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:39.116158 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:39.116312 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:39.120710 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:39.120793 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:39.120879 19673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:28:39.126451 19673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:39.127610 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:39.128661 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:39.138676 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.138680 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.138676 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:39.138744 19673 server_base.cc:1034] running on GCE node
I20260619 22:28:39.139380 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:39.140161 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:39.141341 19673 hybrid_clock.cc:648] HybridClock initialized: now 1781908119141317 us; error 36 us; skew 500 ppm
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:39.144186 19673 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:28:39.145159 19673 webserver.cc:460] Webserver started at http://127.5.176.193:43183/ using document root <none> and password file <none>
I20260619 22:28:39.145751 19673 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:39.145846 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:39.146082 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:39.148200 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "547ffc48b3d0463f8d7903712b1033a3"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "cae80ef87da82ea750c34f2344c534c1"
server_key_iv: "1f73174fded7642fb3af6c0013aee418"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.148792 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "547ffc48b3d0463f8d7903712b1033a3"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "cae80ef87da82ea750c34f2344c534c1"
server_key_iv: "1f73174fded7642fb3af6c0013aee418"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.152629 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 22:28:39.155118 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.156455 19673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:28:39.156598 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "547ffc48b3d0463f8d7903712b1033a3"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "cae80ef87da82ea750c34f2344c534c1"
server_key_iv: "1f73174fded7642fb3af6c0013aee418"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.156716 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:39.189980 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:39.193392 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:39.193670 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:39.194329 19673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:39.195184 19673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:39.195255 19673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.195328 19673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:39.195381 19673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.204319 19673 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:37705
I20260619 22:28:39.204347 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:37705 every 8 connection(s)
I20260619 22:28:39.205435 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:28:39.208246 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19673
I20260619 22:28:39.208401 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:28:39.208730 5827 external_mini_cluster.cc:1428] Setting key e0c224d25782048d7ae965096eef1eeb
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:28:39.220690 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.207411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:36059 (local address 127.5.176.254:45071)
0619 22:28:39.207569 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.207574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.208614 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 22:28:39.210102 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.210105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.210188 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.210267 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:39.211503 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.212042 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.213193 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.213461 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.213578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.216238 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:39.216242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:39.216244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:39.216279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:39.218001 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.218617 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.218621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.218623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.218741 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.219138 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.219141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.219142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.219388 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:39.219512 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.220135 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.220314 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:28:39.221484 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.207865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:36059)
0619 22:28:39.208431 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.208493 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.209871 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.210441 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.210448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.210818 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:39.211346 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.211362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.212175 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.212180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.213065 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.213074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.213620 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.213624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.213793 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.214541 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:39.214558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:39.216056 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:39.218128 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:39.218176 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:39.218195 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:39.218500 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:39.218866 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:39.218869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:39.218871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:39.219023 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:39.219527 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:39.219532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:39.219789 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.220006 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.220417 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260619 22:28:39.223027 19802 heartbeater.cc:347] Connected to a master server at 127.5.176.254:45071
I20260619 22:28:39.223326 19802 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:39.223887 19802 heartbeater.cc:511] Master 127.5.176.254:45071 requested a full tablet report, sending...
I20260619 22:28:39.225615 19610 ts_manager.cc:194] Registered new tserver with Master: 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705)
I20260619 22:28:39.226886 19610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:36059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:28:39.237859 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:45071
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:39.347416 19807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:39.347716 19807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:39.347795 19807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:39.351735 19807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:39.351832 19807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:39.351927 19807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:28:39.357116 19807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:39.358251 19807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:39.359300 19807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:39.369414 19813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.369413 19812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.369453 19815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:39.369748 19807 server_base.cc:1034] running on GCE node
I20260619 22:28:39.370219 19807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:39.370922 19807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:39.372097 19807 hybrid_clock.cc:648] HybridClock initialized: now 1781908119372082 us; error 28 us; skew 500 ppm
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:39.375593 19807 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:28:39.376612 19807 webserver.cc:460] Webserver started at http://127.5.176.194:39933/ using document root <none> and password file <none>
I20260619 22:28:39.377211 19807 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:39.377306 19807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:39.377547 19807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:39.379760 19807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "4c1542e7938e44f6aaface3036bff165"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "030e2e7e0f9992f3f804964ccb2bb613"
server_key_iv: "4f286b18ab0324d315954c57a6f39b83"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.380362 19807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "4c1542e7938e44f6aaface3036bff165"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "030e2e7e0f9992f3f804964ccb2bb613"
server_key_iv: "4f286b18ab0324d315954c57a6f39b83"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.384287 19807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:28:39.386811 19822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.388072 19807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:28:39.388217 19807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "4c1542e7938e44f6aaface3036bff165"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "030e2e7e0f9992f3f804964ccb2bb613"
server_key_iv: "4f286b18ab0324d315954c57a6f39b83"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.388330 19807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:39.405054 19807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:39.408434 19807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:39.408641 19807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:39.409248 19807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:39.410075 19807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:39.410144 19807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.410269 19807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:39.410320 19807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.419353 19807 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:36381
I20260619 22:28:39.419379 19935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:36381 every 8 connection(s)
I20260619 22:28:39.420367 19807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:28:39.424688 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19807
I20260619 22:28:39.424818 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:28:39.425098 5827 external_mini_cluster.cc:1428] Setting key 2924045425b3b8d9d22ebc66e1019c39
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:28:39.434705 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.422289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:54651 (local address 127.5.176.254:45071)
0619 22:28:39.422447 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.422452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.423358 (+ 906us) server_negotiation.cc:407] Connection header received
0619 22:28:39.424659 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.424665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.424729 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.424841 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:39.426251 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.426775 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.427947 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.428163 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.428270 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.431045 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:39.431049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:39.431051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:39.431075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:39.432521 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.433049 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.433052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.433054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.433134 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.433481 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.433484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.433486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.433668 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:39.433783 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.434432 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.434534 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:28:39.435618 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.422739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:54651)
0619 22:28:39.423194 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.423249 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.424409 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.425084 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.425091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.425571 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:39.426082 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.426096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.426906 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.426910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.427795 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.427804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.428319 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.428323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.428504 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.429116 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:39.429132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:39.430862 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:39.432656 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:39.432662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:39.432678 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:39.432941 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:39.433223 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:39.433228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:39.433230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:39.433372 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:39.433811 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:39.433818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:39.434185 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.434369 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.434713 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:28:39.436803 19936 heartbeater.cc:347] Connected to a master server at 127.5.176.254:45071
I20260619 22:28:39.437062 19936 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:39.437685 19936 heartbeater.cc:511] Master 127.5.176.254:45071 requested a full tablet report, sending...
I20260619 22:28:39.438920 19610 ts_manager.cc:194] Registered new tserver with Master: 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381)
I20260619 22:28:39.439797 19610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:54651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:28:39.451061 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:45071
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:39.563364 19941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:39.563632 19941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:39.563674 19941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:39.567817 19941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:39.567891 19941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:39.567952 19941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:28:39.573782 19941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:39.574846 19941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:39.575822 19941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:39.585417 19946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.585491 19949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.585685 19947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:39.585690 19941 server_base.cc:1034] running on GCE node
I20260619 22:28:39.586126 19941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:39.586884 19941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:39.588075 19941 hybrid_clock.cc:648] HybridClock initialized: now 1781908119588062 us; error 35 us; skew 500 ppm
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:39.590785 19941 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:28:39.591660 19941 webserver.cc:460] Webserver started at http://127.5.176.195:37513/ using document root <none> and password file <none>
I20260619 22:28:39.592208 19941 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:39.592269 19941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:39.592482 19941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:39.594558 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "8d8ec059b79d49febcfdbbcae3280baf"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "9f3555add5826a3c72cd83346aaccb6d"
server_key_iv: "d56a42e94879cc07e56de5ae385ace1a"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.595110 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "8d8ec059b79d49febcfdbbcae3280baf"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "9f3555add5826a3c72cd83346aaccb6d"
server_key_iv: "d56a42e94879cc07e56de5ae385ace1a"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.599004 19941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:28:39.601573 19956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.602962 19941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:28:39.603068 19941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "8d8ec059b79d49febcfdbbcae3280baf"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "9f3555add5826a3c72cd83346aaccb6d"
server_key_iv: "d56a42e94879cc07e56de5ae385ace1a"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.603196 19941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:39.617033 19941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:39.620227 19941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:39.620437 19941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:39.621047 19941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:39.621918 19941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:39.621977 19941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.622040 19941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:39.622072 19941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.631414 19941 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42237
I20260619 22:28:39.631430 20069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42237 every 8 connection(s)
I20260619 22:28:39.632352 19941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:28:39.637434 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 19941
I20260619 22:28:39.637557 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:28:39.637823 5827 external_mini_cluster.cc:1428] Setting key b51f7f87ffa8401658e7a91e4086e147
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:39.648820 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.634231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:49411 (local address 127.5.176.254:45071)
0619 22:28:39.634353 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.634357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.635214 (+ 857us) server_negotiation.cc:407] Connection header received
0619 22:28:39.636527 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.636530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.636587 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.636666 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:39.638177 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.638992 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.640962 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.641327 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.641427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.644174 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:39.644178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:39.644181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:39.644208 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:39.646301 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.647032 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.647036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.647038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.647150 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.647657 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.647661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.647662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.647876 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:39.648000 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.648531 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.648651 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:28:39.649650 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.634616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:49411)
0619 22:28:39.635062 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.635105 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.636308 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.636834 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.636841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.637271 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:39.637971 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.637992 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.639152 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.639155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.640817 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.640827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.641471 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.641475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.641656 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.642316 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:39.642332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:39.644018 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:39.646479 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:39.646485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:39.646504 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:39.646868 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:39.647300 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:39.647305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:39.647312 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:39.647532 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:39.647998 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:39.648005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:39.648268 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.648489 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.648822 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260619 22:28:39.650808 20070 heartbeater.cc:347] Connected to a master server at 127.5.176.254:45071
I20260619 22:28:39.651054 20070 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:39.651660 20070 heartbeater.cc:511] Master 127.5.176.254:45071 requested a full tablet report, sending...
I20260619 22:28:39.652712 19610 ts_manager.cc:194] Registered new tserver with Master: 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
I20260619 22:28:39.653301 19610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:49411
I20260619 22:28:39.663971 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:28:39.674491 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.666108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.5.176.254:45071)
0619 22:28:39.666296 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.666302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.666472 (+ 170us) server_negotiation.cc:407] Connection header received
0619 22:28:39.666641 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.666644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.666685 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.666767 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:39.667494 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.668138 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.668980 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.669211 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.669320 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.670786 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:39.670790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:39.670793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:39.670825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:39.672346 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.672815 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.672824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.672825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.672960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:39.673319 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:39.673322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:39.673323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:39.673587 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:39.673700 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.674103 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.674289 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 22:28:39.677263 19610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:28:39.680305 19610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:28:39.695288 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.689508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.5.176.195:42237)
0619 22:28:39.690361 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.690376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.690401 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:28:39.690477 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.690485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.690657 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.690845 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:39.691044 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.692003 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.693541 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.694336 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.694449 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.694595 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.695021 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.695121 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":97,"threads_started":1}
I20260619 22:28:39.695281 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.688767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:37705 (local address 127.0.0.1:40074)
0619 22:28:39.689395 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.689456 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.689592 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.690491 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.690497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.690521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:39.690694 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.690701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.692192 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.692195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.693667 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.693679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.694818 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.694821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.694946 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.694980 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.695053 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.695120 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":77,"thread_start_us":182,"threads_started":1}
I20260619 22:28:39.695335 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.688926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.5.176.193:37705)
0619 22:28:39.689940 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.689956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.689982 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:28:39.690070 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.690075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.690314 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.690500 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:39.690858 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.692027 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.693815 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.694675 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.694905 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.695053 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.695101 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.695191 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":94,"threads_started":1}
I20260619 22:28:39.695669 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.689504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.5.176.194:36381)
0619 22:28:39.689939 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.689953 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.690549 (+ 596us) server_negotiation.cc:407] Connection header received
0619 22:28:39.690622 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.690626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.690783 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.690908 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:39.691316 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.692418 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.693894 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.694640 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.694771 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.694927 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.695197 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.695281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":94,"threads_started":1}
I20260619 22:28:39.695813 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.688781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42237 (local address 127.0.0.1:50580)
0619 22:28:39.689981 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.689996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.690084 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.690795 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.690798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.690820 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:39.690943 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.690951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.692124 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.692127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.693402 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.693419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.694476 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.694481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.694610 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.694656 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.694769 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.695674 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":49,"thread_start_us":56,"threads_started":1}
I20260619 22:28:39.696072 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.689045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:36381 (local address 127.0.0.1:41640)
0619 22:28:39.690327 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.690340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.690452 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.691064 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.691067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.691087 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:39.691176 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.691182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.692581 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.692584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.693740 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.693751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.694775 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.694779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.694881 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.694914 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.695000 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.695069 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260619 22:28:39.698848 20004 tablet_service.cc:1467] Processing CreateTablet for tablet ee793856bb0f4e369fdce29c66df9156 (DEFAULT_TABLE table=test-table [id=f1ed1b80837645b5b79d4e5fcd426a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:39.698863 19736 tablet_service.cc:1467] Processing CreateTablet for tablet ee793856bb0f4e369fdce29c66df9156 (DEFAULT_TABLE table=test-table [id=f1ed1b80837645b5b79d4e5fcd426a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:39.698869 19870 tablet_service.cc:1467] Processing CreateTablet for tablet ee793856bb0f4e369fdce29c66df9156 (DEFAULT_TABLE table=test-table [id=f1ed1b80837645b5b79d4e5fcd426a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:39.699937 19870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee793856bb0f4e369fdce29c66df9156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:39.699937 19736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee793856bb0f4e369fdce29c66df9156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:39.699965 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee793856bb0f4e369fdce29c66df9156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:39.706059 20087 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Bootstrap starting.
I20260619 22:28:39.707713 20089 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Bootstrap starting.
I20260619 22:28:39.708385 20088 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap starting.
I20260619 22:28:39.708915 20087 tablet_bootstrap.cc:654] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:39.709770 20087 log.cc:824] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:39.710425 20089 tablet_bootstrap.cc:654] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:39.710610 20088 tablet_bootstrap.cc:654] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:39.711346 20089 log.cc:824] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:39.711829 20087 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: No bootstrap required, opened a new log
I20260619 22:28:39.711822 20088 log.cc:824] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:39.712044 20087 ts_tablet_manager.cc:1397] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:28:39.713368 20089 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: No bootstrap required, opened a new log
I20260619 22:28:39.713523 20088 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: No bootstrap required, opened a new log
I20260619 22:28:39.713568 20089 ts_tablet_manager.cc:1397] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:28:39.713713 20088 ts_tablet_manager.cc:1397] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:28:39.716048 20087 raft_consensus.cc:348] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.716358 20087 raft_consensus.cc:374] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:39.716410 20087 raft_consensus.cc:729] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8ec059b79d49febcfdbbcae3280baf, State: Initialized, Role: FOLLOWER
I20260619 22:28:39.716677 20088 raft_consensus.cc:348] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.716863 20087 consensus_queue.cc:260] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.716955 20088 raft_consensus.cc:374] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:39.717021 20088 raft_consensus.cc:729] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547ffc48b3d0463f8d7903712b1033a3, State: Initialized, Role: FOLLOWER
I20260619 22:28:39.717338 20089 raft_consensus.cc:348] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.717509 20088 consensus_queue.cc:260] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.717631 20089 raft_consensus.cc:374] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:39.717679 20089 raft_consensus.cc:729] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1542e7938e44f6aaface3036bff165, State: Initialized, Role: FOLLOWER
I20260619 22:28:39.717880 20087 ts_tablet_manager.cc:1428] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:28:39.718154 20089 consensus_queue.cc:260] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.718287 20070 heartbeater.cc:503] Master 127.5.176.254:45071 was elected leader, sending a full tablet report...
I20260619 22:28:39.718585 20088 ts_tablet_manager.cc:1428] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:28:39.718828 19936 heartbeater.cc:503] Master 127.5.176.254:45071 was elected leader, sending a full tablet report...
I20260619 22:28:39.718886 19802 heartbeater.cc:503] Master 127.5.176.254:45071 was elected leader, sending a full tablet report...
I20260619 22:28:39.719116 20089 ts_tablet_manager.cc:1428] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 22:28:39.733352 20095 raft_consensus.cc:482] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:39.733608 20095 raft_consensus.cc:504] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.734845 20095 leader_election.cc:288] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
I20260619 22:28:39.738962 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.735036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:37705 (local address 127.5.176.194:52355)
0619 22:28:39.735234 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.735248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.735341 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.735699 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.735702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.735723 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:39.735860 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.735867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.736799 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.736802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.737671 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.737680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.738509 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.738512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.738624 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.738654 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.738737 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.738815 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"spinlock_wait_cycles":3456}
I20260619 22:28:39.739014 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.735154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52355 (local address 127.5.176.193:37705)
0619 22:28:39.735287 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.735292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.735308 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:28:39.735502 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.735506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.735575 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.735673 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:39.735981 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.736700 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.737798 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.738386 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.738508 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.738562 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.738762 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.738861 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:28:39.739737 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.735238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:45337 (local address 127.5.176.195:42237)
0619 22:28:39.735390 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.735395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.735680 (+ 285us) server_negotiation.cc:407] Connection header received
0619 22:28:39.735845 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.735848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.735901 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.735988 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:39.736272 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.736977 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.738083 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.738730 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.738851 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.738902 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.739537 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.739617 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:28:39.740154 19756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee793856bb0f4e369fdce29c66df9156" candidate_uuid: "4c1542e7938e44f6aaface3036bff165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547ffc48b3d0463f8d7903712b1033a3" is_pre_election: true
I20260619 22:28:39.740257 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.735148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42237 (local address 127.5.176.194:45337)
0619 22:28:39.735577 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.735591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:39.735751 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:39.736018 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:39.736020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:39.736038 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:39.736159 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.736165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.737089 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.737091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.737944 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:39.737954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.738975 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.738979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:39.739374 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.739403 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:28:39.740037 (+ 634us) client_negotiation.cc:240] Negotiation successful
0619 22:28:39.740112 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":3,"thread_start_us":146,"threads_started":1}
I20260619 22:28:39.740526 19756 raft_consensus.cc:2454] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1542e7938e44f6aaface3036bff165 in term 0.
I20260619 22:28:39.740903 20024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee793856bb0f4e369fdce29c66df9156" candidate_uuid: "4c1542e7938e44f6aaface3036bff165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8ec059b79d49febcfdbbcae3280baf" is_pre_election: true
I20260619 22:28:39.741039 19824 leader_election.cc:302] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1542e7938e44f6aaface3036bff165, 547ffc48b3d0463f8d7903712b1033a3; no voters:
I20260619 22:28:39.741202 20024 raft_consensus.cc:2454] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1542e7938e44f6aaface3036bff165 in term 0.
I20260619 22:28:39.741393 20095 raft_consensus.cc:2781] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:28:39.741495 20095 raft_consensus.cc:482] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:28:39.741542 20095 raft_consensus.cc:3037] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:39.742653 20095 raft_consensus.cc:504] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.743016 20095 leader_election.cc:288] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [CANDIDATE]: Term 1 election: Requested vote from peers 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
I20260619 22:28:39.743366 19756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee793856bb0f4e369fdce29c66df9156" candidate_uuid: "4c1542e7938e44f6aaface3036bff165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547ffc48b3d0463f8d7903712b1033a3"
I20260619 22:28:39.743369 20024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee793856bb0f4e369fdce29c66df9156" candidate_uuid: "4c1542e7938e44f6aaface3036bff165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8ec059b79d49febcfdbbcae3280baf"
I20260619 22:28:39.743489 20024 raft_consensus.cc:3037] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:39.743489 19756 raft_consensus.cc:3037] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:39.744740 19756 raft_consensus.cc:2454] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1542e7938e44f6aaface3036bff165 in term 1.
I20260619 22:28:39.744740 20024 raft_consensus.cc:2454] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1542e7938e44f6aaface3036bff165 in term 1.
I20260619 22:28:39.745050 19826 leader_election.cc:302] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1542e7938e44f6aaface3036bff165, 8d8ec059b79d49febcfdbbcae3280baf; no voters:
I20260619 22:28:39.745239 20095 raft_consensus.cc:2781] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:39.745427 20095 raft_consensus.cc:686] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1542e7938e44f6aaface3036bff165, State: Running, Role: LEADER
I20260619 22:28:39.745738 20095 consensus_queue.cc:237] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:39.749054 19610 catalog_manager.cc:5495] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1542e7938e44f6aaface3036bff165 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "4c1542e7938e44f6aaface3036bff165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } health_report { overall_health: HEALTHY } } }
I20260619 22:28:39.773185 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.769306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.5.176.194:36381)
0619 22:28:39.769437 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:39.769441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:39.769456 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:28:39.769525 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:39.769527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:39.769580 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:39.769678 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:28:39.769944 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.770500 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.771309 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:39.771610 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:39.771720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:39.772139 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:28:39.772642 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:28:39.772734 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:39.772944 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 22:28:39.773009 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:28:39.779649 19756 raft_consensus.cc:1261] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Refusing update from remote peer 4c1542e7938e44f6aaface3036bff165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:39.779649 20024 raft_consensus.cc:1261] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Refusing update from remote peer 4c1542e7938e44f6aaface3036bff165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:39.780526 20097 consensus_queue.cc:1035] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:39.780743 20095 consensus_queue.cc:1035] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:39.790166 20101 mvcc.cc:205] Tried to move back new op lower bound from 7298695658609233920 to 7298695658481262592. Current Snapshot: MvccSnapshot[applied={T|T < 7298695658609233920}]
I20260619 22:28:39.792943 20102 mvcc.cc:205] Tried to move back new op lower bound from 7298695658609233920 to 7298695658481262592. Current Snapshot: MvccSnapshot[applied={T|T < 7298695658609233920 or (T in {7298695658609233920})}]
I20260619 22:28:39.797238 19610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:39.797422 19610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:39.799853 19610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Sending DeleteTablet for 3 replicas of tablet ee793856bb0f4e369fdce29c66df9156
I20260619 22:28:39.800681 19736 tablet_service.cc:1514] Processing DeleteTablet for tablet ee793856bb0f4e369fdce29c66df9156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:39 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:40074
I20260619 22:28:39.800691 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet ee793856bb0f4e369fdce29c66df9156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:39 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:50580
I20260619 22:28:39.801229 20110 tablet_replica.cc:330] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: stopping tablet replica
I20260619 22:28:39.801220 19870 tablet_service.cc:1514] Processing DeleteTablet for tablet ee793856bb0f4e369fdce29c66df9156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:39 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:41640
I20260619 22:28:39.801570 20110 raft_consensus.cc:2229] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:39.801756 20112 tablet_replica.cc:330] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: stopping tablet replica
I20260619 22:28:39.801779 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19673
I20260619 22:28:39.801923 20110 raft_consensus.cc:2258] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:39.803810 20110 ts_tablet_manager.cc:1905] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:39.804235 20112 raft_consensus.cc:2229] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:28:39.804648 20112 raft_consensus.cc:2258] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:39.806191 20112 ts_tablet_manager.cc:1905] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:39.807703 20110 ts_tablet_manager.cc:1918] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:39.807812 20110 log.cc:1196] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/ee793856bb0f4e369fdce29c66df9156
I20260619 22:28:39.808161 20110 ts_tablet_manager.cc:1939] T ee793856bb0f4e369fdce29c66df9156 P 8d8ec059b79d49febcfdbbcae3280baf: Deleting consensus metadata
W20260619 22:28:39.809405 19596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:37705 (error 108)
I20260619 22:28:39.809530 19598 catalog_manager.cc:4841] TS 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): tablet ee793856bb0f4e369fdce29c66df9156 (table test-table [id=f1ed1b80837645b5b79d4e5fcd426a4d]) successfully deleted
I20260619 22:28:39.809473 20112 ts_tablet_manager.cc:1918] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:39.809758 20112 log.cc:1196] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/ee793856bb0f4e369fdce29c66df9156
I20260619 22:28:39.810105 20112 ts_tablet_manager.cc:1939] T ee793856bb0f4e369fdce29c66df9156 P 4c1542e7938e44f6aaface3036bff165: Deleting consensus metadata
I20260619 22:28:39.810250 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19807
I20260619 22:28:39.812047 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.811674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:37705 (local address 127.0.0.1:40078)
0619 22:28:39.811853 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.811934 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 22:28:39.812225 19596 catalog_manager.cc:4584] TS 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee793856bb0f4e369fdce29c66df9156: Network error: Client connection negotiation failed: client connection to 127.5.176.193:37705: connect: Connection refused (error 111)
I20260619 22:28:39.816592 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19941
I20260619 22:28:39.816851 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:39.816539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:36381 (local address 127.0.0.1:41650)
0619 22:28:39.816675 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:39.816743 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:36381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 22:28:39.822850 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 19579
I20260619 22:28:39.829654 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45071
--webserver_interface=127.5.176.254
--webserver_port=37247
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:39.940105 20114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:39.940375 20114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:39.940416 20114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:39.944519 20114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:39.944581 20114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:39.944622 20114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:39.944639 20114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:39.944687 20114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:39.949987 20114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=37247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:39.951155 20114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:39.952203 20114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:39.966198 20121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.966261 20123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:39.966280 20120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:39.966264 20114 server_base.cc:1034] running on GCE node
I20260619 22:28:39.967149 20114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:39.968408 20114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:39.969612 20114 hybrid_clock.cc:648] HybridClock initialized: now 1781908119969584 us; error 48 us; skew 500 ppm
Jun 19 22:28:39 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:39.972630 20114 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:39.973524 20114 webserver.cc:460] Webserver started at http://127.5.176.254:37247/ using document root <none> and password file <none>
I20260619 22:28:39.974079 20114 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:39.974210 20114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:39.978216 20114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 22:28:39.980526 20130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:39.981801 20114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:28:39.981910 20114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "9698070a3daa4366b1a852dae52beac6"
format_stamp: "Formatted at 2026-06-19 22:28:38 on dist-test-slave-bt27"
server_key: "2080382f48537c94e1c6eeacb0520951"
server_key_iv: "541fd4f4e1514d9f4cdbe3cd1ac51acc"
server_key_version: "encryptionkey@0"
I20260619 22:28:39.982652 20114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:39.993580 20114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:39.997154 20114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:39.998449 20114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:40.010483 20114 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:45071
I20260619 22:28:40.010802 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:45071 every 8 connection(s)
I20260619 22:28:40.011926 20114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:40.016758 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20114
I20260619 22:28:40.017289 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:37705
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:45071
--webserver_port=43183
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:40.018113 20183 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:28:40.020773 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap starting.
I20260619 22:28:40.047537 20183 log.cc:824] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:40.059228 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:28:40.060102 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap complete.
I20260619 22:28:40.065933 20183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:40.066500 20183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9698070a3daa4366b1a852dae52beac6, State: Initialized, Role: FOLLOWER
I20260619 22:28:40.067044 20183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:40.067210 20183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:40.067304 20183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:40.067433 20183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:28:40.069761 20183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:40.070209 20183 leader_election.cc:302] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9698070a3daa4366b1a852dae52beac6; no voters:
I20260619 22:28:40.070595 20183 leader_election.cc:288] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:28:40.070745 20187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:28:40.071008 20187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 LEADER]: Becoming Leader. State: Replica: 9698070a3daa4366b1a852dae52beac6, State: Running, Role: LEADER
I20260619 22:28:40.071338 20187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:40.071954 20183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:40.072420 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9698070a3daa4366b1a852dae52beac6. Latest consensus state: current_term: 2 leader_uuid: "9698070a3daa4366b1a852dae52beac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } } }
I20260619 22:28:40.072536 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:40.072988 20191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:40.072952 20188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9698070a3daa4366b1a852dae52beac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } } }
I20260619 22:28:40.073184 20188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:40.078300 20191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:40.078965 20191 catalog_manager.cc:1259] Loaded cluster ID: 8679da619e51439aa43fec2d2e2438ad
I20260619 22:28:40.079325 20191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:40.080786 20191 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 22:28:40.086215 20191 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 22:28:40.092954 20191 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:40.093910 20191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Loaded TSK: 0
I20260619 22:28:40.095271 20191 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:28:40.160419 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:40.160756 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:40.160843 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:40.165701 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:40.165802 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:40.165897 20185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:28:40.171818 20185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:37705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=43183
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:40.173091 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:40.174365 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:40.184909 20211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:40.184900 20210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:40.184900 20213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:40.186091 20185 server_base.cc:1034] running on GCE node
I20260619 22:28:40.186694 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:40.187433 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:40.188633 20185 hybrid_clock.cc:648] HybridClock initialized: now 1781908120188596 us; error 58 us; skew 500 ppm
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:40.191900 20185 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:28:40.193176 20185 webserver.cc:460] Webserver started at http://127.5.176.193:43183/ using document root <none> and password file <none>
I20260619 22:28:40.193977 20185 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:40.194092 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:40.199486 20185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:28:40.202466 20220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.204133 20185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:28:40.204265 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "547ffc48b3d0463f8d7903712b1033a3"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "cae80ef87da82ea750c34f2344c534c1"
server_key_iv: "1f73174fded7642fb3af6c0013aee418"
server_key_version: "encryptionkey@0"
I20260619 22:28:40.204980 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:40.229108 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:40.233104 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:40.233306 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:40.233992 20185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:40.235298 20227 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:28:40.239636 20185 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:28:40.239729 20185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 22:28:40.239807 20185 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:28:40.241551 20185 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:28:40.241602 20185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 22:28:40.242019 20227 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap starting.
I20260619 22:28:40.252353 20185 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:37705
I20260619 22:28:40.252410 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:37705 every 8 connection(s)
I20260619 22:28:40.253376 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:28:40.256307 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20185
I20260619 22:28:40.256757 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:36381
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:45071
--webserver_port=39933
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:40.261595 20227 log.cc:824] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:40.270427 20227 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:28:40.270907 20227 tablet_bootstrap.cc:492] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap complete.
I20260619 22:28:40.272284 20227 ts_tablet_manager.cc:1397] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260619 22:28:40.275174 20227 raft_consensus.cc:348] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:40.275713 20227 raft_consensus.cc:729] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547ffc48b3d0463f8d7903712b1033a3, State: Initialized, Role: FOLLOWER
I20260619 22:28:40.275966 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.255534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:50381)
0619 22:28:40.256146 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.256218 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.257536 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.259209 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.259221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.259700 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:40.260164 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.260179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.261980 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.261986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.263665 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.263681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.264726 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.264733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.265110 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.265898 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:40.265926 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:40.268142 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:40.271512 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.271519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.271535 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.272043 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.272490 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.272494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.272496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.272729 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.274059 (+ 1330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:40.274066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:40.274408 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.274614 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.274970 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":155,"threads_started":1}
I20260619 22:28:40.276290 20227 consensus_queue.cc:260] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } }
I20260619 22:28:40.276961 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.255534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:50381 (local address 127.5.176.254:45071)
0619 22:28:40.255927 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.255947 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.256509 (+ 562us) server_negotiation.cc:407] Connection header received
0619 22:28:40.258234 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.258249 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.258700 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.259122 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:40.260358 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.261790 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.263816 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.264541 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.264729 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.268425 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:40.268433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:40.268449 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:40.268486 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:40.271318 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.272178 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.272186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.272191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.272354 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.272894 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.272899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.272901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.273913 (+ 1012us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:40.274059 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.274767 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.275112 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":119,"threads_started":1}
I20260619 22:28:40.277374 20227 ts_tablet_manager.cc:1428] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 22:28:40.279340 20335 heartbeater.cc:347] Connected to a master server at 127.5.176.254:45071
I20260619 22:28:40.279604 20335 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:40.280120 20335 heartbeater.cc:511] Master 127.5.176.254:45071 requested a full tablet report, sending...
I20260619 22:28:40.282708 20147 ts_manager.cc:194] Registered new tserver with Master: 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705)
I20260619 22:28:40.286118 20147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:50381
I20260619 22:28:40.288646 20335 heartbeater.cc:503] Master 127.5.176.254:45071 was elected leader, sending a full tablet report...
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908119, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:28:40.296929 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.285685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.5.176.193:37705)
0619 22:28:40.286688 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.286703 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.286731 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:28:40.286812 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.286818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.287038 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.287190 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:40.287925 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.289062 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.290389 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.290704 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.290785 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.293364 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:40.293379 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:40.293389 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:40.293411 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:40.295053 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.295543 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.295549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.295552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.295630 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.295917 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.295921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.295923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.296152 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:40.296290 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.296619 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.296777 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":112,"threads_started":1}
I20260619 22:28:40.296929 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.285603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:37705 (local address 127.0.0.1:40080)
0619 22:28:40.286246 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.286290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.286417 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.287208 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.287215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.287660 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:40.287782 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.287791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.289232 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.289236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.290231 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.290244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.290828 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.290840 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.290958 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.291614 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:40.291632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:40.293213 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:40.295183 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.295192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.295197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.295429 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.295725 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.295729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.295731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.295817 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.296263 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:40.296269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:40.296395 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.296532 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.296728 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":149,"threads_started":1}
I20260619 22:28:40.298087 20269 tablet_service.cc:1514] Processing DeleteTablet for tablet ee793856bb0f4e369fdce29c66df9156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:39 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:40080
I20260619 22:28:40.299190 20344 tablet_replica.cc:330] stopping tablet replica
I20260619 22:28:40.299511 20344 raft_consensus.cc:2229] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:40.299765 20344 raft_consensus.cc:2258] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:40.300756 20344 ts_tablet_manager.cc:1905] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:40.304368 20344 ts_tablet_manager.cc:1918] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:40.304492 20344 log.cc:1196] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/ee793856bb0f4e369fdce29c66df9156
I20260619 22:28:40.304775 20344 ts_tablet_manager.cc:1939] T ee793856bb0f4e369fdce29c66df9156 P 547ffc48b3d0463f8d7903712b1033a3: Deleting consensus metadata
I20260619 22:28:40.306788 20132 catalog_manager.cc:4841] TS 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705): tablet ee793856bb0f4e369fdce29c66df9156 (table test-table [id=f1ed1b80837645b5b79d4e5fcd426a4d]) successfully deleted
W20260619 22:28:40.381745 20339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:40.382035 20339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:40.382083 20339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:40.386485 20339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:40.386565 20339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:40.386686 20339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:28:40.392390 20339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:36381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=39933
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:40.393507 20339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:40.394660 20339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:40.404832 20349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:40.404841 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:40.404841 20350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:40.405036 20339 server_base.cc:1034] running on GCE node
I20260619 22:28:40.405630 20339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:40.406350 20339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:40.407537 20339 hybrid_clock.cc:648] HybridClock initialized: now 1781908120407521 us; error 31 us; skew 500 ppm
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:40.410614 20339 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:28:40.411550 20339 webserver.cc:460] Webserver started at http://127.5.176.194:39933/ using document root <none> and password file <none>
I20260619 22:28:40.412094 20339 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:40.412164 20339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:40.416133 20339 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:28:40.418381 20359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.419754 20339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:28:40.419864 20339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "4c1542e7938e44f6aaface3036bff165"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "030e2e7e0f9992f3f804964ccb2bb613"
server_key_iv: "4f286b18ab0324d315954c57a6f39b83"
server_key_version: "encryptionkey@0"
I20260619 22:28:40.420447 20339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:40.451555 20339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:40.454967 20339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:40.455137 20339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:40.455710 20339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:40.456513 20339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:40.456571 20339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.456614 20339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:40.456630 20339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.466277 20339 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:36381
I20260619 22:28:40.466400 20472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:36381 every 8 connection(s)
I20260619 22:28:40.467350 20339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:28:40.475253 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20339
I20260619 22:28:40.475733 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:42237
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:45071
--webserver_port=37513
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:40.483681 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.469350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:45533 (local address 127.5.176.254:45071)
0619 22:28:40.469497 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.469500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.470502 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 22:28:40.471858 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.471862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.471927 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.472009 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:40.473418 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.473975 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.475258 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.475631 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.475753 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.478801 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:40.478808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:40.478814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:40.478853 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:40.481044 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.481676 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.481683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.481688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.481792 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.482216 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.482221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.482224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.482517 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:40.482650 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.483233 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.483405 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:28:40.484390 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.469757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:45533)
0619 22:28:40.470333 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.470386 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.471629 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.472270 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.472278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.472675 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:40.473218 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.473261 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.474169 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.474172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.475129 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.475139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.475825 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.475830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.476086 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.476763 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:40.476781 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:40.478584 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:40.481210 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.481216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.481231 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.481544 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.481902 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.481906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.481907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.482074 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.482664 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:40.482670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:40.482955 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.483126 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.483429 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260619 22:28:40.486078 20473 heartbeater.cc:347] Connected to a master server at 127.5.176.254:45071
I20260619 22:28:40.486414 20473 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:40.487136 20473 heartbeater.cc:511] Master 127.5.176.254:45071 requested a full tablet report, sending...
I20260619 22:28:40.488461 20147 ts_manager.cc:194] Registered new tserver with Master: 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381)
I20260619 22:28:40.489082 20147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:45533
W20260619 22:28:40.599686 20476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:40.600024 20476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:40.600109 20476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:40.604470 20476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:40.604573 20476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:40.604691 20476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:28:40.610383 20476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:42237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=37513
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:40.611568 20476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:40.612645 20476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:40.622751 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:40.622758 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:40.622754 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:40.622970 20476 server_base.cc:1034] running on GCE node
I20260619 22:28:40.623605 20476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:40.624322 20476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:40.625555 20476 hybrid_clock.cc:648] HybridClock initialized: now 1781908120625507 us; error 63 us; skew 500 ppm
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:40.628518 20476 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:28:40.629511 20476 webserver.cc:460] Webserver started at http://127.5.176.195:37513/ using document root <none> and password file <none>
I20260619 22:28:40.630102 20476 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:40.630242 20476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:40.634282 20476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:28:40.636498 20491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.637814 20476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:28:40.637949 20476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "8d8ec059b79d49febcfdbbcae3280baf"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "9f3555add5826a3c72cd83346aaccb6d"
server_key_iv: "d56a42e94879cc07e56de5ae385ace1a"
server_key_version: "encryptionkey@0"
I20260619 22:28:40.638590 20476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:40.652014 20476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:40.655862 20476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:40.656101 20476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:40.656765 20476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:40.657655 20476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:40.657729 20476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.657805 20476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:40.657850 20476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:40.667415 20476 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42237
I20260619 22:28:40.667438 20604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42237 every 8 connection(s)
I20260619 22:28:40.668443 20476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:28:40.673619 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20476
Jun 19 22:28:40 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:40.682895 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.674374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.5.176.254:45071)
0619 22:28:40.674656 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.674661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.674765 (+ 104us) server_negotiation.cc:407] Connection header received
0619 22:28:40.674917 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.674920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.674982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.675078 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:40.675797 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.676308 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.677160 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.677391 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.677499 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.679120 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:40.679123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:40.679125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:40.679147 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:40.680954 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.681470 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.681473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.681474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.681549 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.681851 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.681854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.681856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.682193 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:40.682349 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.682581 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.682720 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":1280,"thread_start_us":102,"threads_started":1}
I20260619 22:28:40.683714 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.670380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:33793 (local address 127.5.176.254:45071)
0619 22:28:40.670522 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.670525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.671332 (+ 807us) server_negotiation.cc:407] Connection header received
0619 22:28:40.672740 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.672746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.672809 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.672897 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:40.674455 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.675035 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.676182 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.676407 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.676523 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.679404 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:40.679410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:40.679416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:40.679462 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:40.681211 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.681851 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.681854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.681856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.681990 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:40.682431 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:40.682433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:40.682434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:40.682639 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:40.682731 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.683453 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.683579 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":14464}
I20260619 22:28:40.684828 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.670737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:33793)
0619 22:28:40.671175 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.671217 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.672486 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.673074 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.673082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.673565 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:40.674291 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.674312 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.675168 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.675172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.676055 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.676064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.676559 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.676563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.676739 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.677341 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:40.677356 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:40.679230 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:40.681333 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.681339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.681356 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.681729 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.682086 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:40.682088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:40.682090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:40.682314 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:40.682761 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:40.682769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:40.683133 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.683358 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.683708 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260619 22:28:40.685889 20605 heartbeater.cc:347] Connected to a master server at 127.5.176.254:45071
I20260619 22:28:40.686115 20605 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:40.686640 20605 heartbeater.cc:511] Master 127.5.176.254:45071 requested a full tablet report, sending...
I20260619 22:28:40.687655 20147 ts_manager.cc:194] Registered new tserver with Master: 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
I20260619 22:28:40.688249 20147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:33793
I20260619 22:28:40.696885 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:28:40.701570 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.697597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.5.176.254:45071)
0619 22:28:40.697729 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.697733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.697942 (+ 209us) server_negotiation.cc:407] Connection header received
0619 22:28:40.698091 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.698093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.698262 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.698352 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:28:40.698579 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.699092 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.700035 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.700254 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.700339 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.700723 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:28:40.701178 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:28:40.701279 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.701393 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.701448 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:28:40.702375 20147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:28:40.705788 20147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:28:40.714713 20269 tablet_service.cc:1467] Processing CreateTablet for tablet ada2bab443a3405db095ad0dbb64a121 (DEFAULT_TABLE table=test-table [id=9b21c42318ac411f913449b52e1f5af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:40.715446 20269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada2bab443a3405db095ad0dbb64a121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:40.718931 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.713616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:36381 (local address 127.0.0.1:41656)
0619 22:28:40.713785 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.713799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.713886 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.714710 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.714713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.714735 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:40.714849 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.714856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.716220 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.716223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.717458 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.717474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.718429 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.718433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.718531 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.718599 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.718683 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.718779 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 22:28:40.719056 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.713771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.5.176.194:36381)
0619 22:28:40.714123 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.714172 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.714197 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:28:40.714287 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.714298 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.714555 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.714688 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:40.714993 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.716072 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.717609 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.718291 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.718408 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.718592 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.718732 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.718866 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":151,"threads_started":1}
I20260619 22:28:40.719161 20227 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap starting.
I20260619 22:28:40.720152 20227 tablet_bootstrap.cc:654] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:40.721685 20227 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: No bootstrap required, opened a new log
I20260619 22:28:40.721793 20227 ts_tablet_manager.cc:1397] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 22:28:40.722293 20227 raft_consensus.cc:348] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.722421 20227 raft_consensus.cc:374] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:40.722460 20227 raft_consensus.cc:729] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547ffc48b3d0463f8d7903712b1033a3, State: Initialized, Role: FOLLOWER
I20260619 22:28:40.722658 20227 consensus_queue.cc:260] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.722357 20407 tablet_service.cc:1467] Processing CreateTablet for tablet ada2bab443a3405db095ad0dbb64a121 (DEFAULT_TABLE table=test-table [id=9b21c42318ac411f913449b52e1f5af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:40.723616 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada2bab443a3405db095ad0dbb64a121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:40.723739 20227 ts_tablet_manager.cc:1428] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 22:28:40.724149 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.713793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42237 (local address 127.0.0.1:50596)
0619 22:28:40.714260 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.714273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.714354 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.719827 (+ 5473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.719831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.719850 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:40.719941 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.719947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.721329 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.721332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.722469 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.722483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.723677 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.723681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.723779 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.723808 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.723879 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.723991 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":110,"threads_started":1}
I20260619 22:28:40.724179 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.714238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.5.176.195:42237)
0619 22:28:40.719106 (+ 4868us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.719121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.719149 (+ 28us) server_negotiation.cc:407] Connection header received
0619 22:28:40.719237 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.719242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.719481 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.719654 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:40.720082 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.721176 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.722662 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.723536 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.723653 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.723808 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.723926 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.724028 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4727,"thread_start_us":147,"threads_started":1}
I20260619 22:28:40.727576 20539 tablet_service.cc:1467] Processing CreateTablet for tablet ada2bab443a3405db095ad0dbb64a121 (DEFAULT_TABLE table=test-table [id=9b21c42318ac411f913449b52e1f5af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:40.728802 20539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada2bab443a3405db095ad0dbb64a121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:40.732740 20618 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Bootstrap starting.
I20260619 22:28:40.735535 20618 tablet_bootstrap.cc:654] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:40.736330 20618 log.cc:824] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:40.737205 20620 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Bootstrap starting.
I20260619 22:28:40.738780 20618 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: No bootstrap required, opened a new log
I20260619 22:28:40.738971 20618 ts_tablet_manager.cc:1397] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:28:40.740067 20620 tablet_bootstrap.cc:654] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:40.740962 20620 log.cc:824] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:40.743249 20620 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: No bootstrap required, opened a new log
I20260619 22:28:40.743089 20618 raft_consensus.cc:348] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.743382 20618 raft_consensus.cc:374] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:40.743440 20618 raft_consensus.cc:729] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1542e7938e44f6aaface3036bff165, State: Initialized, Role: FOLLOWER
I20260619 22:28:40.743460 20620 ts_tablet_manager.cc:1397] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 22:28:40.743942 20618 consensus_queue.cc:260] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.744673 20473 heartbeater.cc:503] Master 127.5.176.254:45071 was elected leader, sending a full tablet report...
I20260619 22:28:40.745061 20618 ts_tablet_manager.cc:1428] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:28:40.746865 20620 raft_consensus.cc:348] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.747150 20620 raft_consensus.cc:374] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:40.747226 20620 raft_consensus.cc:729] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8ec059b79d49febcfdbbcae3280baf, State: Initialized, Role: FOLLOWER
I20260619 22:28:40.747730 20620 consensus_queue.cc:260] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.748646 20605 heartbeater.cc:503] Master 127.5.176.254:45071 was elected leader, sending a full tablet report...
I20260619 22:28:40.748835 20620 ts_tablet_manager.cc:1428] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 22:28:40.755038 20336 tablet.cc:2369] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:28:40.791196 20341 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:40.791458 20341 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.792717 20341 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
I20260619 22:28:40.797147 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.793076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47175 (local address 127.5.176.195:42237)
0619 22:28:40.793313 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.793318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.793367 (+ 49us) server_negotiation.cc:407] Connection header received
0619 22:28:40.793630 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.793633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.793685 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.793747 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:40.794095 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.794654 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.795860 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.796491 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.796629 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.796685 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.796931 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.797020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260619 22:28:40.797384 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.793006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:36381 (local address 127.5.176.193:43201)
0619 22:28:40.793397 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.793413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.793517 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.793830 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.793833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.793876 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:40.794004 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.794009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.794815 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.794818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.795759 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.795770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.796976 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.796979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.797063 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.797084 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.797155 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.797217 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":67,"spinlock_wait_cycles":2688,"thread_start_us":83,"threads_started":1}
I20260619 22:28:40.797406 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.793152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:43201 (local address 127.5.176.194:36381)
0619 22:28:40.793322 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.793328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.793501 (+ 173us) server_negotiation.cc:407] Connection header received
0619 22:28:40.793654 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.793656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.793700 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.793800 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:40.794175 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.794676 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.795918 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.796511 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.796629 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.796685 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.797196 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.797282 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 22:28:40.797672 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.792972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42237 (local address 127.5.176.193:47175)
0619 22:28:40.793273 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.793290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:40.793440 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:40.793795 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:40.793797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:40.793813 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:40.793983 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.793989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.794767 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.794772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.795734 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:40.795743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.796637 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.796644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:40.796756 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.796820 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 22:28:40.796905 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 22:28:40.797000 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":43,"spinlock_wait_cycles":34560,"thread_start_us":90,"threads_started":1}
I20260619 22:28:40.798130 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada2bab443a3405db095ad0dbb64a121" candidate_uuid: "547ffc48b3d0463f8d7903712b1033a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1542e7938e44f6aaface3036bff165" is_pre_election: true
I20260619 22:28:40.798334 20559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada2bab443a3405db095ad0dbb64a121" candidate_uuid: "547ffc48b3d0463f8d7903712b1033a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8ec059b79d49febcfdbbcae3280baf" is_pre_election: true
I20260619 22:28:40.798556 20427 raft_consensus.cc:2454] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547ffc48b3d0463f8d7903712b1033a3 in term 0.
I20260619 22:28:40.798578 20559 raft_consensus.cc:2454] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547ffc48b3d0463f8d7903712b1033a3 in term 0.
I20260619 22:28:40.799017 20221 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1542e7938e44f6aaface3036bff165, 547ffc48b3d0463f8d7903712b1033a3; no voters:
I20260619 22:28:40.799268 20341 raft_consensus.cc:2781] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:28:40.799371 20341 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:28:40.799432 20341 raft_consensus.cc:3037] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:40.800482 20341 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.800859 20341 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
I20260619 22:28:40.801271 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada2bab443a3405db095ad0dbb64a121" candidate_uuid: "547ffc48b3d0463f8d7903712b1033a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1542e7938e44f6aaface3036bff165"
I20260619 22:28:40.801286 20559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada2bab443a3405db095ad0dbb64a121" candidate_uuid: "547ffc48b3d0463f8d7903712b1033a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8ec059b79d49febcfdbbcae3280baf"
I20260619 22:28:40.801399 20427 raft_consensus.cc:3037] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:40.801430 20559 raft_consensus.cc:3037] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:40.802563 20559 raft_consensus.cc:2454] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547ffc48b3d0463f8d7903712b1033a3 in term 1.
I20260619 22:28:40.802611 20427 raft_consensus.cc:2454] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547ffc48b3d0463f8d7903712b1033a3 in term 1.
I20260619 22:28:40.802873 20224 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547ffc48b3d0463f8d7903712b1033a3, 8d8ec059b79d49febcfdbbcae3280baf; no voters:
I20260619 22:28:40.803090 20341 raft_consensus.cc:2781] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:40.803290 20341 raft_consensus.cc:686] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 LEADER]: Becoming Leader. State: Replica: 547ffc48b3d0463f8d7903712b1033a3, State: Running, Role: LEADER
I20260619 22:28:40.803580 20341 consensus_queue.cc:237] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:40.806833 20146 catalog_manager.cc:5495] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193). New cstate: current_term: 1 leader_uuid: "547ffc48b3d0463f8d7903712b1033a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } health_report { overall_health: HEALTHY } } }
I20260619 22:28:40.826280 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.822081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.5.176.193:37705)
0619 22:28:40.822432 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:40.822443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:40.822456 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:28:40.822542 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:40.822544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:40.822604 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:40.822747 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:28:40.823029 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.823574 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.824562 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:40.824818 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:40.824955 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:40.825272 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:28:40.825767 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:28:40.825921 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:40.825989 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 22:28:40.826060 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":139,"threads_started":1}
I20260619 22:28:40.830670 20427 raft_consensus.cc:1261] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Refusing update from remote peer 547ffc48b3d0463f8d7903712b1033a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:40.830688 20559 raft_consensus.cc:1261] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Refusing update from remote peer 547ffc48b3d0463f8d7903712b1033a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:40.831506 20626 consensus_queue.cc:1035] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:40.831768 20341 consensus_queue.cc:1035] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:40.839774 20631 mvcc.cc:205] Tried to move back new op lower bound from 7298695662914981888 to 7298695662814023680. Current Snapshot: MvccSnapshot[applied={T|T < 7298695662914981888}]
I20260619 22:28:40.840760 20632 mvcc.cc:205] Tried to move back new op lower bound from 7298695662914981888 to 7298695662814023680. Current Snapshot: MvccSnapshot[applied={T|T < 7298695662914981888}]
I20260619 22:28:40.848769 20147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:40.848999 20147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:40.851725 20147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Sending DeleteTablet for 3 replicas of tablet ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:40.852618 20407 tablet_service.cc:1514] Processing DeleteTablet for tablet ada2bab443a3405db095ad0dbb64a121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:40 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:41656
I20260619 22:28:40.852859 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 20185
I20260619 22:28:40.852838 20539 tablet_service.cc:1514] Processing DeleteTablet for tablet ada2bab443a3405db095ad0dbb64a121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:40 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:50596
I20260619 22:28:40.853852 20640 tablet_replica.cc:330] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: stopping tablet replica
I20260619 22:28:40.854297 20640 raft_consensus.cc:2229] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:40.854740 20640 raft_consensus.cc:2258] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:40.855738 20641 tablet_replica.cc:330] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: stopping tablet replica
I20260619 22:28:40.856087 20641 raft_consensus.cc:2229] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:40.856408 20641 raft_consensus.cc:2258] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:40.856511 20640 ts_tablet_manager.cc:1905] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:40.858013 20641 ts_tablet_manager.cc:1905] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:40.859761 20640 ts_tablet_manager.cc:1918] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 22:28:40.859779 20132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:37705 (error 108)
I20260619 22:28:40.859887 20640 log.cc:1196] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:40.860272 20640 ts_tablet_manager.cc:1939] T ada2bab443a3405db095ad0dbb64a121 P 4c1542e7938e44f6aaface3036bff165: Deleting consensus metadata
I20260619 22:28:40.860705 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 20339
I20260619 22:28:40.860962 20641 ts_tablet_manager.cc:1918] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:40.861054 20641 log.cc:1196] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:40.861338 20641 ts_tablet_manager.cc:1939] T ada2bab443a3405db095ad0dbb64a121 P 8d8ec059b79d49febcfdbbcae3280baf: Deleting consensus metadata
I20260619 22:28:40.862320 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.861813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:37705 (local address 127.0.0.1:40104)
0619 22:28:40.862066 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.862227 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 22:28:40.862634 20134 catalog_manager.cc:4841] TS 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): tablet ada2bab443a3405db095ad0dbb64a121 (table test-table [id=9b21c42318ac411f913449b52e1f5af9]) successfully deleted
W20260619 22:28:40.862732 20132 catalog_manager.cc:4584] TS 547ffc48b3d0463f8d7903712b1033a3 (127.5.176.193:37705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ada2bab443a3405db095ad0dbb64a121: Network error: Client connection negotiation failed: client connection to 127.5.176.193:37705: connect: Connection refused (error 111)
I20260619 22:28:40.867462 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 20476
I20260619 22:28:40.869484 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:40.867681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:36381 (local address 127.0.0.1:41664)
0619 22:28:40.869272 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:40.869352 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:36381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1472}
I20260619 22:28:40.873894 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 20114
I20260619 22:28:40.880579 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45071
--webserver_interface=127.5.176.254
--webserver_port=37247
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:40.997041 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:40.997355 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:40.997397 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:41.001619 20643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:41.001708 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:41.001729 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:41.001746 20643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:41.001762 20643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:41.007566 20643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=37247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:41.008692 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:41.009687 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:41.018658 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.018658 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.018668 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:41.018953 20643 server_base.cc:1034] running on GCE node
I20260619 22:28:41.019658 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:41.020898 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:41.022085 20643 hybrid_clock.cc:648] HybridClock initialized: now 1781908121022069 us; error 30 us; skew 500 ppm
Jun 19 22:28:41 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:41.025024 20643 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:41.026010 20643 webserver.cc:460] Webserver started at http://127.5.176.254:37247/ using document root <none> and password file <none>
I20260619 22:28:41.026634 20643 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:41.026700 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:41.030879 20643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:28:41.033262 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.034796 20643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:28:41.034965 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "9698070a3daa4366b1a852dae52beac6"
format_stamp: "Formatted at 2026-06-19 22:28:38 on dist-test-slave-bt27"
server_key: "2080382f48537c94e1c6eeacb0520951"
server_key_iv: "541fd4f4e1514d9f4cdbe3cd1ac51acc"
server_key_version: "encryptionkey@0"
I20260619 22:28:41.035596 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:41.058127 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:41.061764 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:41.061995 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:41.069797 20643 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:45071
I20260619 22:28:41.070188 20710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:45071 every 8 connection(s)
I20260619 22:28:41.071063 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:41.075618 20711 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:28:41.077409 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap starting.
I20260619 22:28:41.078127 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20643
I20260619 22:28:41.078652 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:37705
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:45071
--webserver_port=43183
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:41.096561 20711 log.cc:824] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:41.108397 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:28:41.108872 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6: Bootstrap complete.
I20260619 22:28:41.113137 20711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:41.113577 20711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9698070a3daa4366b1a852dae52beac6, State: Initialized, Role: FOLLOWER
I20260619 22:28:41.114190 20711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:41.114338 20711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:41.114434 20711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:41.114573 20711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:28:41.116750 20711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:41.117151 20711 leader_election.cc:302] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9698070a3daa4366b1a852dae52beac6; no voters:
I20260619 22:28:41.117532 20711 leader_election.cc:288] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 22:28:41.117622 20715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 22:28:41.117815 20715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [term 3 LEADER]: Becoming Leader. State: Replica: 9698070a3daa4366b1a852dae52beac6, State: Running, Role: LEADER
I20260619 22:28:41.118175 20715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } }
I20260619 22:28:41.118911 20711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:41.119207 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9698070a3daa4366b1a852dae52beac6. Latest consensus state: current_term: 3 leader_uuid: "9698070a3daa4366b1a852dae52beac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } } }
I20260619 22:28:41.119338 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:41.119758 20716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9698070a3daa4366b1a852dae52beac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698070a3daa4366b1a852dae52beac6" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 45071 } } }
I20260619 22:28:41.119922 20716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698070a3daa4366b1a852dae52beac6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:41.120993 20719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:41.127215 20719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:41.127957 20719 catalog_manager.cc:1259] Loaded cluster ID: 8679da619e51439aa43fec2d2e2438ad
I20260619 22:28:41.128024 20719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 22:28:41.129509 20719 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f86a7d7e9ad google::LogMessage::Fail() at ??:0
@ 0x7f86a7d8188a google::LogMessage::SendToLog() at ??:0
@ 0x7f86a7d7e4de google::LogMessage::Flush() at ??:0
@ 0x7f86a7d7fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f86b26c1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f86b26c2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f86b26bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f86b2703640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f86a9fd03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f86a9fd0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f86a9fd23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f86a9fc1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f86ab6056db start_thread at ??:0
@ 0x7f86a687c61f clone at ??:0
W20260619 22:28:41.240906 20713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:41.241321 20713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:41.241400 20713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:41.247817 20713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:41.247926 20713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:41.248019 20713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:28:41.255872 20713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:37705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=43183
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:41.257051 20713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:41.258095 20713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:41.268687 20752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.268697 20755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.268740 20753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:41.269220 20713 server_base.cc:1034] running on GCE node
I20260619 22:28:41.269690 20713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:41.270524 20713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:41.271701 20713 hybrid_clock.cc:648] HybridClock initialized: now 1781908121271684 us; error 31 us; skew 500 ppm
Jun 19 22:28:41 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:41.274887 20713 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:28:41.276000 20713 webserver.cc:460] Webserver started at http://127.5.176.193:43183/ using document root <none> and password file <none>
I20260619 22:28:41.276602 20713 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:41.276695 20713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:41.281023 20713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:28:41.283402 20762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.284809 20713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:28:41.284947 20713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "547ffc48b3d0463f8d7903712b1033a3"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "cae80ef87da82ea750c34f2344c534c1"
server_key_iv: "1f73174fded7642fb3af6c0013aee418"
server_key_version: "encryptionkey@0"
I20260619 22:28:41.285521 20713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:41.298870 20713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:41.302310 20713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:41.302516 20713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:41.303210 20713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:41.304564 20769 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:28:41.308943 20713 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:28:41.309034 20713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 22:28:41.309119 20713 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:28:41.310992 20713 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:28:41.311054 20713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 22:28:41.311311 20769 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap starting.
I20260619 22:28:41.322108 20713 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:37705
I20260619 22:28:41.323550 20713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:28:41.325363 20876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:37705 every 8 connection(s)
I20260619 22:28:41.327102 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20713
I20260619 22:28:41.327701 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:36381
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:45071
--webserver_port=39933
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:28:41.334393 20769 log.cc:824] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:41.344885 20769 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:28:41.345356 20769 tablet_bootstrap.cc:492] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Bootstrap complete.
I20260619 22:28:41.346837 20769 ts_tablet_manager.cc:1397] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.011s
I20260619 22:28:41.350396 20769 raft_consensus.cc:348] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:41.350973 20769 raft_consensus.cc:729] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547ffc48b3d0463f8d7903712b1033a3, State: Initialized, Role: FOLLOWER
I20260619 22:28:41.351508 20769 consensus_queue.cc:260] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:41.352619 20769 ts_tablet_manager.cc:1428] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 22:28:41.381165 20879 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:28:41.327866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:36597)
0619 22:28:41.329273 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.329336 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:41.330643 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:41.380362 (+ 49719us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":148,"threads_started":1}
W20260619 22:28:41.381814 20877 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:45071 (0 consecutive failures): Network error: Failed to ping master at 127.5.176.254:45071: Client connection negotiation failed: client connection to 127.5.176.254:45071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:28:41.382628 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.382348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:54213)
0619 22:28:41.382492 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.382566 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 22:28:41.383368 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.383126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:38357)
0619 22:28:41.383251 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.383306 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:28:41.383569 20877 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:28:41.451972 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:41.452275 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:41.452322 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:41.456821 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:41.456902 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:41.456965 20880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:28:41.462641 20880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:36381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=39933
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:41.464170 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:41.465396 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:41.475787 20890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.475796 20888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.475787 20887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:41.476217 20880 server_base.cc:1034] running on GCE node
I20260619 22:28:41.476682 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:41.477386 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:41.478562 20880 hybrid_clock.cc:648] HybridClock initialized: now 1781908121478538 us; error 33 us; skew 500 ppm
Jun 19 22:28:41 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:41.481509 20880 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:28:41.482466 20880 webserver.cc:460] Webserver started at http://127.5.176.194:39933/ using document root <none> and password file <none>
I20260619 22:28:41.483011 20880 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:41.483073 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:41.487135 20880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:28:41.489395 20897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.490922 20880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:28:41.491072 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "4c1542e7938e44f6aaface3036bff165"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "030e2e7e0f9992f3f804964ccb2bb613"
server_key_iv: "4f286b18ab0324d315954c57a6f39b83"
server_key_version: "encryptionkey@0"
I20260619 22:28:41.491662 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:41.508190 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:41.511814 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:41.512007 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:41.512553 20880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:41.513386 20880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:41.513432 20880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.513473 20880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:41.513489 20880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.522557 20880 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:36381
I20260619 22:28:41.522605 21010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:36381 every 8 connection(s)
I20260619 22:28:41.523602 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:28:41.527516 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 20880
I20260619 22:28:41.527595 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.525801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:60825)
0619 22:28:41.526297 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.526411 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260619 22:28:41.528003 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:42237
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:45071
--webserver_port=37513
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:41.528090 21011 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:45071 (0 consecutive failures): Network error: Failed to ping master at 127.5.176.254:45071: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
I20260619 22:28:41.528715 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.528479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:53639)
0619 22:28:41.528603 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.528654 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 22:28:41.529548 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.529281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:36839)
0619 22:28:41.529413 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.529471 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 22:28:41.529795 21011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:28:41.641070 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:41.641407 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:41.641497 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:41.647210 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:41.647297 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:41.647392 21014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:28:41.655509 21014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:42237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=37513
--tserver_master_addrs=127.5.176.254:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:41.656709 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:41.657783 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:41.668156 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.668180 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:41.668170 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:41.668319 21014 server_base.cc:1034] running on GCE node
I20260619 22:28:41.668983 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:41.669708 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:41.670949 21014 hybrid_clock.cc:648] HybridClock initialized: now 1781908121670933 us; error 28 us; skew 500 ppm
Jun 19 22:28:41 dist-test-slave-bt27 krb5kdc[19566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:41.674355 21014 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:28:41.675441 21014 webserver.cc:460] Webserver started at http://127.5.176.195:37513/ using document root <none> and password file <none>
I20260619 22:28:41.676015 21014 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:41.676080 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:41.680270 21014 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:28:41.682499 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.683895 21014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:28:41.684044 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "8d8ec059b79d49febcfdbbcae3280baf"
format_stamp: "Formatted at 2026-06-19 22:28:39 on dist-test-slave-bt27"
server_key: "9f3555add5826a3c72cd83346aaccb6d"
server_key_iv: "d56a42e94879cc07e56de5ae385ace1a"
server_key_version: "encryptionkey@0"
I20260619 22:28:41.684680 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:41.722627 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:41.726171 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:41.726405 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:41.727057 21014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:41.727924 21014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:41.727973 21014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.728046 21014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:41.728085 21014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:41.737485 21014 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:42237
I20260619 22:28:41.737525 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:42237 every 8 connection(s)
I20260619 22:28:41.738574 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:28:41.742861 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.740818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:52811)
0619 22:28:41.741299 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.741420 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
W20260619 22:28:41.743427 21143 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:45071 (0 consecutive failures): Network error: Failed to ping master at 127.5.176.254:45071: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
I20260619 22:28:41.744107 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.743861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:60539)
0619 22:28:41.743994 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.744048 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 22:28:41.744848 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:41.744649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:41537)
0619 22:28:41.744756 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:41.744800 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:28:41.745102 21143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 22:28:41.745676 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21014
I20260619 22:28:42.385134 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.384416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:43801)
0619 22:28:42.384900 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:42.384981 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":79,"threads_started":1}
I20260619 22:28:42.531205 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.530667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:56997)
0619 22:28:42.531010 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:42.531081 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":115,"threads_started":1}
I20260619 22:28:42.746341 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.745760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:42813)
0619 22:28:42.746094 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:42.746232 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260619 22:28:42.831369 21150 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:42.831722 21150 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:42.833164 21150 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
Jun 19 22:28:42 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.195@KRBTEST.COM
Jun 19 22:28:42 dist-test-slave-bt27 krb5kdc[19566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.194@KRBTEST.COM
I20260619 22:28:42.850445 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.833795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:34599 (local address 127.5.176.194:36381)
0619 22:28:42.834237 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:42.834259 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:42.834489 (+ 230us) server_negotiation.cc:407] Connection header received
0619 22:28:42.835309 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:42.835317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:42.835802 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:42.836236 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:42.837656 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.838755 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.840130 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.840677 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.840884 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:42.844623 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:42.844629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:42.844645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:42.844675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:42.846892 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:42.847636 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:42.847642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:42.847646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:42.847770 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:42.848330 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:42.848335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:42.848336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:42.848954 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:42.849110 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:42.849802 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 22:28:42.850197 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260619 22:28:42.850430 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.833522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:36381 (local address 127.5.176.193:34599)
0619 22:28:42.834337 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:42.834352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:42.834470 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:42.836203 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:42.836207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:42.836819 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:42.837423 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:42.837441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.838906 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.838909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:42.839983 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:42.839992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.840842 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.840846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:42.841028 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:42.841782 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:42.841816 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:42.844367 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:42.847042 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:42.847044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:42.847046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:42.847474 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:42.847983 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:42.847986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:42.847990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:42.848181 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:42.849199 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:42.849202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:42.849416 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 22:28:42.849667 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 22:28:42.850117 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":700,"spinlock_wait_cycles":7552,"thread_start_us":85,"threads_started":1}
I20260619 22:28:42.850533 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.833522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:42237 (local address 127.5.176.193:47915)
0619 22:28:42.833758 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:42.833778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:42.833939 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:42.836121 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:42.836129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:42.836843 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:42.837425 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:42.837451 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.839036 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.839039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:42.839986 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:42.839995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.840779 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.840786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:42.841006 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:42.841827 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:42.841841 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:42.843955 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:42.846852 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:42.846862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:42.846879 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:42.847304 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:42.847958 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:42.847962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:42.847964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:42.848161 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:42.849117 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:42.849127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:42.849418 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 22:28:42.849667 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 22:28:42.850258 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":242,"spinlock_wait_cycles":13312}
I20260619 22:28:42.851274 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:42.833770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47915 (local address 127.5.176.195:42237)
0619 22:28:42.834254 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:42.834283 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:42.834335 (+ 52us) server_negotiation.cc:407] Connection header received
0619 22:28:42.835228 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:42.835235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:42.835668 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:42.835972 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:42.837684 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.838884 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.840136 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:42.840640 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:42.840819 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:42.844250 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:42.844259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:42.844283 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:42.844323 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:42.846700 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:42.847532 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:42.847543 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:42.847553 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:42.847804 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:42.848306 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:42.848309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:42.848311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:42.848954 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:42.849109 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:42.849937 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 22:28:42.850356 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":126,"threads_started":1}
W20260619 22:28:42.854045 20766 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
W20260619 22:28:42.854310 20763 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:42.854410 20763 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 547ffc48b3d0463f8d7903712b1033a3; no voters: 4c1542e7938e44f6aaface3036bff165, 8d8ec059b79d49febcfdbbcae3280baf
I20260619 22:28:42.854702 21150 raft_consensus.cc:2726] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:28:43.386615 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:43.386114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:37807)
0619 22:28:43.386439 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:43.386511 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
I20260619 22:28:43.532480 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:43.532069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:33773)
0619 22:28:43.532324 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:43.532380 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260619 22:28:43.747670 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:43.747244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:42273)
0619 22:28:43.747513 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:43.747575 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260619 22:28:44.387821 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:44.387386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:39971)
0619 22:28:44.387672 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:44.387727 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
I20260619 22:28:44.533618 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:44.533203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:33337)
0619 22:28:44.533472 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:44.533529 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260619 22:28:44.748759 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:44.748406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:42017)
0619 22:28:44.748627 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:44.748674 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260619 22:28:44.978650 21160 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:44.978786 21160 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:44.979213 21160 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
W20260619 22:28:44.980038 20763 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
W20260619 22:28:44.980131 20766 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:44.980257 20766 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 547ffc48b3d0463f8d7903712b1033a3; no voters: 4c1542e7938e44f6aaface3036bff165, 8d8ec059b79d49febcfdbbcae3280baf
I20260619 22:28:44.980406 21160 raft_consensus.cc:2726] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:28:45.389304 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:45.388730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:52391)
0619 22:28:45.389047 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:45.389179 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260619 22:28:45.535074 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:45.534578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:50793)
0619 22:28:45.534884 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:45.534949 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":77,"threads_started":1}
I20260619 22:28:45.750294 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:45.749675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:60713)
0619 22:28:45.750048 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:45.750133 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":105,"threads_started":1}
I20260619 22:28:46.391036 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:46.390489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:57969)
0619 22:28:46.390848 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:46.390912 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":104,"threads_started":1}
I20260619 22:28:46.536695 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:46.536074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:50129)
0619 22:28:46.536466 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:46.536572 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":131,"threads_started":1}
I20260619 22:28:46.752000 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:46.751459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:38209)
0619 22:28:46.751815 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:46.751906 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":127,"threads_started":1}
I20260619 22:28:47.392454 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:47.391891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:35121)
0619 22:28:47.392250 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:47.392330 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20260619 22:28:47.538216 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:47.537572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:35279)
0619 22:28:47.537945 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:47.538028 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260619 22:28:47.753396 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:47.752894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:33071)
0619 22:28:47.753217 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:47.753294 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
I20260619 22:28:48.199689 21170 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:48.199849 21170 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:48.200320 21170 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
W20260619 22:28:48.201195 20763 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
W20260619 22:28:48.201354 20766 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:48.201463 20766 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 547ffc48b3d0463f8d7903712b1033a3; no voters: 4c1542e7938e44f6aaface3036bff165, 8d8ec059b79d49febcfdbbcae3280baf
I20260619 22:28:48.201658 21170 raft_consensus.cc:2726] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:28:48.393985 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:48.393423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:52661)
0619 22:28:48.393767 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:48.393849 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260619 22:28:48.539701 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:48.539189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:34583)
0619 22:28:48.539520 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:48.539593 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260619 22:28:48.755085 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:48.754371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:50193)
0619 22:28:48.754836 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:48.754940 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":87,"threads_started":1}
I20260619 22:28:49.395449 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:49.394873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:32959)
0619 22:28:49.395231 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:49.395321 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260619 22:28:49.541208 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:49.540675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:39599)
0619 22:28:49.541006 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:49.541082 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260619 22:28:49.756476 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:49.755905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:51633)
0619 22:28:49.756275 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:49.756350 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":80,"threads_started":1}
I20260619 22:28:50.396888 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:50.396327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:56643)
0619 22:28:50.396695 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:50.396778 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
I20260619 22:28:50.542964 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:50.542260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:33325)
0619 22:28:50.542709 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:50.542793 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
I20260619 22:28:50.758122 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:50.757527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:53703)
0619 22:28:50.757873 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:50.757989 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260619 22:28:51.008749 21180 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:51.008929 21180 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:51.009549 21180 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
W20260619 22:28:51.010519 20763 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
W20260619 22:28:51.010629 20766 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:51.010777 20766 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 547ffc48b3d0463f8d7903712b1033a3; no voters: 4c1542e7938e44f6aaface3036bff165, 8d8ec059b79d49febcfdbbcae3280baf
I20260619 22:28:51.010934 21180 raft_consensus.cc:2726] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:28:51.398670 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:51.398061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:41413)
0619 22:28:51.398484 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:51.398565 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":155,"threads_started":1}
I20260619 22:28:51.544534 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:51.544050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:36457)
0619 22:28:51.544357 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:51.544427 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
I20260619 22:28:51.759703 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:51.759130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:37301)
0619 22:28:51.759452 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:51.759535 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":84,"threads_started":1}
I20260619 22:28:52.400158 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:52.399580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:40769)
0619 22:28:52.399935 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:52.400015 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":128,"threads_started":1}
I20260619 22:28:52.546026 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:52.545434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:38039)
0619 22:28:52.545802 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:52.545893 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":82,"threads_started":1}
I20260619 22:28:52.761153 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:52.760652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:50737)
0619 22:28:52.760951 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:52.761029 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260619 22:28:52.921955 21187 raft_consensus.cc:482] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:52.922132 21187 raft_consensus.cc:504] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1542e7938e44f6aaface3036bff165" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 36381 } } peers { permanent_uuid: "8d8ec059b79d49febcfdbbcae3280baf" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 42237 } } peers { permanent_uuid: "547ffc48b3d0463f8d7903712b1033a3" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 37705 } }
I20260619 22:28:52.922770 21187 leader_election.cc:288] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381), 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237)
W20260619 22:28:52.923686 20766 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d8ec059b79d49febcfdbbcae3280baf (127.5.176.195:42237): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
W20260619 22:28:52.923846 20763 leader_election.cc:341] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c1542e7938e44f6aaface3036bff165 (127.5.176.194:36381): Not found: Tablet not found: ada2bab443a3405db095ad0dbb64a121
I20260619 22:28:52.923979 20763 leader_election.cc:302] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 547ffc48b3d0463f8d7903712b1033a3; no voters: 4c1542e7938e44f6aaface3036bff165, 8d8ec059b79d49febcfdbbcae3280baf
I20260619 22:28:52.924198 21187 raft_consensus.cc:2726] T ada2bab443a3405db095ad0dbb64a121 P 547ffc48b3d0463f8d7903712b1033a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 22:28:53.401521 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:53.400939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:60707)
0619 22:28:53.401304 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:53.401391 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":83,"threads_started":1}
I20260619 22:28:53.547665 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:53.547081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:49489)
0619 22:28:53.547445 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:53.547533 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260619 22:28:53.762705 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:53.762234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:47139)
0619 22:28:53.762519 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:53.762579 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260619 22:28:54.402804 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:54.402363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:46065)
0619 22:28:54.402644 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:54.402705 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260619 22:28:54.549117 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:54.548613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:46675)
0619 22:28:54.548946 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:54.549015 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260619 22:28:54.764209 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:54.763666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:57259)
0619 22:28:54.764031 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:54.764105 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260619 22:28:55.404428 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:55.403861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:42189)
0619 22:28:55.404219 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:55.404297 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260619 22:28:55.550788 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:55.550271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:51947)
0619 22:28:55.550605 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:55.550677 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":135,"threads_started":1}
I20260619 22:28:55.765622 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:55.765129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.195:52489)
0619 22:28:55.765429 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:55.765493 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260619 22:28:56.406164 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:56.405587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.193:51829)
0619 22:28:56.405957 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:56.406042 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260619 22:28:56.552407 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:56.551919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:45071 (local address 127.5.176.194:35917)
0619 22:28:56.552212 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:56.552277 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260619 22:28:56.749048 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 20713
I20260619 22:28:56.755424 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 20880
I20260619 22:28:56.761242 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21014
2026-06-19T22:28:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19226 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:28:56 dist-test-slave-bt27 krb5kdc[21201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:28:56 dist-test-slave-bt27 krb5kdc[21201](info): set up 2 sockets
Jun 19 22:28:56 dist-test-slave-bt27 krb5kdc[21201](info): commencing operation
krb5kdc: starting...
W20260619 22:28:57.890532 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:28:57 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:28:58.014425 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37937
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:58.126654 21214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:58.126960 21214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:58.127003 21214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:58.131107 21214 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 22:28:58.131170 21214 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 22:28:58.131186 21214 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 22:28:58.131201 21214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:28:58.131219 21214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:58.131238 21214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:58.131254 21214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:28:58.131270 21214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:28:58.136973 21214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:58.138094 21214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:58.139174 21214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:58.147915 21220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:58.148010 21214 server_base.cc:1034] running on GCE node
W20260619 22:28:58.147907 21219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:58.147907 21222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:58.148850 21214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:58.149883 21214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:58.151046 21214 hybrid_clock.cc:648] HybridClock initialized: now 1781908138151031 us; error 45 us; skew 500 ppm
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:58.157877 21214 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:28:58.158929 21214 webserver.cc:460] Webserver started at http://127.5.176.254:36893/ using document root <none> and password file <none>
I20260619 22:28:58.159513 21214 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:58.159602 21214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:58.159830 21214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:58.161908 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "e293b0e762554081b13d4790e3116e80"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "f714010c13c4601d285430290e2962ab"
server_key_iv: "e600198f11aa2d1b4e1e7378e93e2c79"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.162525 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "e293b0e762554081b13d4790e3116e80"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "f714010c13c4601d285430290e2962ab"
server_key_iv: "e600198f11aa2d1b4e1e7378e93e2c79"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.166323 21214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:28:58.168875 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.170179 21214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:28:58.170341 21214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "e293b0e762554081b13d4790e3116e80"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "f714010c13c4601d285430290e2962ab"
server_key_iv: "e600198f11aa2d1b4e1e7378e93e2c79"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.170461 21214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:58.181705 21214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:58.185207 21214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:58.185441 21214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:58.193181 21214 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:37937
I20260619 22:28:58.193538 21281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:37937 every 8 connection(s)
I20260619 22:28:58.194279 21214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:28:58.197095 21282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:58.201095 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21214
I20260619 22:28:58.201251 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:28:58.201552 5827 external_mini_cluster.cc:1428] Setting key dd3e2b2639ee4a37027e1a0324034881
I20260619 22:28:58.202785 21282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap starting.
I20260619 22:28:58.205144 21282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:58.206005 21282 log.cc:824] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:58.208115 21282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: No bootstrap required, opened a new log
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:58.212824 21282 raft_consensus.cc:348] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:28:58.213076 21282 raft_consensus.cc:374] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:58.213155 21282 raft_consensus.cc:729] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e293b0e762554081b13d4790e3116e80, State: Initialized, Role: FOLLOWER
I20260619 22:28:58.213593 21282 consensus_queue.cc:260] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:28:58.213739 21282 raft_consensus.cc:388] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:28:58.213826 21282 raft_consensus.cc:482] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:28:58.213944 21282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:58.214881 21282 raft_consensus.cc:504] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:28:58.215245 21282 leader_election.cc:302] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e293b0e762554081b13d4790e3116e80; no voters:
I20260619 22:28:58.215610 21282 leader_election.cc:288] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:28:58.215751 21287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:58.215934 21287 raft_consensus.cc:686] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 LEADER]: Becoming Leader. State: Replica: e293b0e762554081b13d4790e3116e80, State: Running, Role: LEADER
I20260619 22:28:58.216284 21287 consensus_queue.cc:237] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:28:58.216964 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.202976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.5.176.254:37937)
0619 22:28:58.203382 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.203399 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.203434 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:28:58.204081 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.204098 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.204422 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.204716 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:58.205670 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.206745 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.207929 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.208477 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.208598 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.211501 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:58.211506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:58.211542 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:58.211579 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:58.213684 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.214294 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.214299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.214303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.214417 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.214711 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.214714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.214716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.215171 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:58.215382 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.215644 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.216085 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":129,"threads_started":1}
I20260619 22:28:58.216975 21282 sys_catalog.cc:564] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:28:58.217490 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e293b0e762554081b13d4790e3116e80. Latest consensus state: current_term: 1 leader_uuid: "e293b0e762554081b13d4790e3116e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } } }
I20260619 22:28:58.217597 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:58.217888 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e293b0e762554081b13d4790e3116e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } } }
I20260619 22:28:58.217976 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: This master's current role is: LEADER
I20260619 22:28:58.219467 21291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:28:58.222275 21291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:28:58.227494 21291 catalog_manager.cc:1347] Generated new cluster ID: c311a325ed1f4585b6d81fb02039c195
I20260619 22:28:58.227578 21291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:28:58.241685 21291 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:28:58.242625 21291 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:28:58.255587 21291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Generated new TSK 0
I20260619 22:28:58.256347 21291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:28:58.293322 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37937
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:58.418231 21308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:58.418565 21308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:58.418609 21308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:58.422608 21308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:58.422662 21308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:58.422720 21308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:28:58.427917 21308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:58.428961 21308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:58.429921 21308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:58.439534 21313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:58.439534 21316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:58.439587 21314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:58.439803 21308 server_base.cc:1034] running on GCE node
I20260619 22:28:58.440219 21308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:58.440927 21308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:58.442190 21308 hybrid_clock.cc:648] HybridClock initialized: now 1781908138442123 us; error 30 us; skew 500 ppm
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:58.445158 21308 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:28:58.446075 21308 webserver.cc:460] Webserver started at http://127.5.176.193:38383/ using document root <none> and password file <none>
I20260619 22:28:58.446684 21308 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:58.446748 21308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:58.446925 21308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:58.449005 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "d0623d78435047efb30d1428779d3962"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "696399bd6972398972dd441ab1d57f22"
server_key_iv: "40fa0a3e36acdf8f9e4f8b9ba1d6f3e7"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.449537 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "d0623d78435047efb30d1428779d3962"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "696399bd6972398972dd441ab1d57f22"
server_key_iv: "40fa0a3e36acdf8f9e4f8b9ba1d6f3e7"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.453363 21308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:28:58.455886 21323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.457258 21308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 22:28:58.457398 21308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "d0623d78435047efb30d1428779d3962"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "696399bd6972398972dd441ab1d57f22"
server_key_iv: "40fa0a3e36acdf8f9e4f8b9ba1d6f3e7"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.457518 21308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:58.476325 21308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:58.479835 21308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:58.480063 21308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:58.480712 21308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:58.481549 21308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:58.481598 21308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.481642 21308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:58.481658 21308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.491698 21308 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:42373
I20260619 22:28:58.491761 21436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:42373 every 8 connection(s)
I20260619 22:28:58.492684 21308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:28:58.500722 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21308
I20260619 22:28:58.500869 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:28:58.501150 5827 external_mini_cluster.cc:1428] Setting key 4349b397435813a358f76e309bff5508
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:58.507354 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.494675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:58259 (local address 127.5.176.254:37937)
0619 22:28:58.494813 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.494817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.495891 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 22:28:58.496735 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.496738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.496794 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.496875 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:58.498024 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.498666 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.499942 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.500167 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.500253 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.502945 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:58.502949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:58.502953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:58.502982 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:58.504651 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.505373 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.505376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.505377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.505466 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.505896 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.505899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.505900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.506121 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:58.506292 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.507016 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.507133 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:28:58.508498 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.495068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:58259)
0619 22:28:58.495727 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:58.495768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:58.496536 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:58.497053 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:58.497059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:58.497394 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:58.497862 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.497875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.498825 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.498828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.499803 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.499814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.500333 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.500337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.500482 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.501123 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:58.501139 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:58.502732 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:58.504791 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:58.504800 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:58.504820 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:58.505246 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:58.505567 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:58.505573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:58.505576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:58.505777 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:58.506298 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:58.506308 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:58.506662 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 22:28:58.506913 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 22:28:58.507309 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":115,"threads_started":1}
I20260619 22:28:58.509897 21437 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37937
I20260619 22:28:58.510207 21437 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:58.510716 21437 heartbeater.cc:511] Master 127.5.176.254:37937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:28:58.512482 21246 ts_manager.cc:194] Registered new tserver with Master: d0623d78435047efb30d1428779d3962 (127.5.176.193:42373)
I20260619 22:28:58.513629 21246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:58259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:28:58.529486 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37937
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:58.640626 21442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:58.640915 21442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:58.640995 21442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:58.645092 21442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:58.645179 21442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:58.645288 21442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:28:58.650547 21442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:58.651557 21442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:58.652540 21442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:58.662052 21448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:58.662093 21447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:58.662209 21442 server_base.cc:1034] running on GCE node
W20260619 22:28:58.662356 21450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:58.662730 21442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:58.663414 21442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:58.664621 21442 hybrid_clock.cc:648] HybridClock initialized: now 1781908138664605 us; error 25 us; skew 500 ppm
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:58.667519 21442 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:28:58.668395 21442 webserver.cc:460] Webserver started at http://127.5.176.194:42665/ using document root <none> and password file <none>
I20260619 22:28:58.668972 21442 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:58.669062 21442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:58.669304 21442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:58.671500 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "14d890684a6728f357f7b1c3d6293697"
server_key_iv: "a690889500d5e9bc47d8164d1cc64276"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.672096 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "14d890684a6728f357f7b1c3d6293697"
server_key_iv: "a690889500d5e9bc47d8164d1cc64276"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.676182 21442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:28:58.678784 21457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.680092 21442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:28:58.680235 21442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "14d890684a6728f357f7b1c3d6293697"
server_key_iv: "a690889500d5e9bc47d8164d1cc64276"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.680373 21442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:58.694799 21442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:58.698052 21442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:58.698298 21442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:58.698941 21442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:58.699769 21442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:58.699839 21442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.699919 21442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:58.699972 21442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.709038 21442 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:40687
I20260619 22:28:58.709076 21570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:40687 every 8 connection(s)
I20260619 22:28:58.709955 21442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:28:58.716037 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21442
I20260619 22:28:58.716156 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:28:58.716468 5827 external_mini_cluster.cc:1428] Setting key 3ef2ba42604d02d97ddd9be9fc031cbd
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:58.724532 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.711820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:40037 (local address 127.5.176.254:37937)
0619 22:28:58.712012 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.712018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.713039 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 22:28:58.713850 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.713853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.713899 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.713976 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:58.715187 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.715727 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.717318 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.717585 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.717696 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.720461 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:58.720469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:58.720471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:58.720496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:58.722009 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.722700 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.722703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.722704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.722785 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.723250 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.723254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.723255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.723473 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:58.723562 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.724277 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.724403 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 22:28:58.725742 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.712198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:40037)
0619 22:28:58.712857 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:58.712923 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:58.713651 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:58.714178 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:58.714188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:58.714554 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:58.715047 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.715060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.715858 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.715861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.717192 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.717202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.717746 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.717750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.718017 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.718725 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:58.718741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:58.720292 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:58.722128 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:58.722155 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:58.722177 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:58.722596 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:58.722887 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:58.722893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:58.722896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:58.723106 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:58.723592 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:58.723602 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:58.723976 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 22:28:58.724202 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 22:28:58.724603 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:28:58.727088 21571 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37937
I20260619 22:28:58.727380 21571 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:58.727901 21571 heartbeater.cc:511] Master 127.5.176.254:37937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:28:58.729079 21246 ts_manager.cc:194] Registered new tserver with Master: 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687)
I20260619 22:28:58.729681 21246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:40037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:28:58.744716 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37937
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:28:58.857348 21576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:28:58.857630 21576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:28:58.857674 21576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:28:58.862094 21576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:28:58.862231 21576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:28:58.862347 21576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:28:58.867811 21576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:28:58.868968 21576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:28:58.870007 21576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:28:58.879633 21582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:58.879673 21581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:28:58.879696 21584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:28:58.879814 21576 server_base.cc:1034] running on GCE node
I20260619 22:28:58.880344 21576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:28:58.880978 21576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:28:58.882203 21576 hybrid_clock.cc:648] HybridClock initialized: now 1781908138882185 us; error 26 us; skew 500 ppm
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:28:58.885177 21576 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:28:58.886116 21576 webserver.cc:460] Webserver started at http://127.5.176.195:34751/ using document root <none> and password file <none>
I20260619 22:28:58.886754 21576 fs_manager.cc:359] Metadata directory not provided
I20260619 22:28:58.886847 21576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:28:58.887144 21576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:28:58.889212 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "e12711d514b141f9b148429bc80980f7"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "35044919b3be98580443517f45d8e70b"
server_key_iv: "1fe589d6bae55e1d37f47aec51bbc67e"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.889786 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "e12711d514b141f9b148429bc80980f7"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "35044919b3be98580443517f45d8e70b"
server_key_iv: "1fe589d6bae55e1d37f47aec51bbc67e"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.893641 21576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:28:58.896435 21591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.897799 21576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:28:58.897984 21576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "e12711d514b141f9b148429bc80980f7"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "35044919b3be98580443517f45d8e70b"
server_key_iv: "1fe589d6bae55e1d37f47aec51bbc67e"
server_key_version: "encryptionkey@0"
I20260619 22:28:58.898118 21576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:28:58.912366 21576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:28:58.915545 21576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:28:58.915755 21576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:28:58.916363 21576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:28:58.917268 21576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:28:58.917335 21576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.917447 21576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:28:58.917486 21576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:28:58.926759 21576 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:34813
I20260619 22:28:58.926779 21704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:34813 every 8 connection(s)
I20260619 22:28:58.927685 21576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:28:58.931202 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21576
I20260619 22:28:58.931346 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:28:58.931633 5827 external_mini_cluster.cc:1428] Setting key 1f2e63339994b2722e697b556ff2cd21
Jun 19 22:28:58 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908138, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:28:58.941146 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.929679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:43187 (local address 127.5.176.254:37937)
0619 22:28:58.929850 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.929854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.930941 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 22:28:58.931807 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.931810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.931869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.931970 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:58.933282 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.933972 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.935172 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.935396 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.935486 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.937948 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:58.937952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:58.937956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:58.937983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:58.939315 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.939769 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.939772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.939773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.939846 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.940164 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.940166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.940167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.940331 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:58.940408 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.940900 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.941014 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:28:58.941905 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.930037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:43187)
0619 22:28:58.930773 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:58.930818 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:58.931616 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:58.932122 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:58.932128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:58.932610 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:28:58.933141 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.933160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.934096 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.934099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.935051 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.935061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.935531 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.935533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.935675 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.936299 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:28:58.936315 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:28:58.937767 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:28:58.939417 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:58.939422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:58.939434 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:58.939683 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:58.939952 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:28:58.939954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:28:58.939956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:28:58.940085 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:28:58.940414 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:28:58.940419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:28:58.940662 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 22:28:58.940822 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 22:28:58.941086 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":142,"threads_started":1}
I20260619 22:28:58.943015 21705 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37937
I20260619 22:28:58.943228 21705 heartbeater.cc:464] Registering TS with master...
I20260619 22:28:58.943662 21705 heartbeater.cc:511] Master 127.5.176.254:37937 requested a full tablet report, sending...
I20260619 22:28:58.944614 21246 ts_manager.cc:194] Registered new tserver with Master: e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813)
I20260619 22:28:58.945278 21246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:43187
I20260619 22:28:58.945461 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:28:58.955078 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.947518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.5.176.254:37937)
0619 22:28:58.947642 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.947646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.947813 (+ 167us) server_negotiation.cc:407] Connection header received
0619 22:28:58.947953 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.947956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.948003 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.948085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:28:58.948761 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.949272 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.950115 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.950371 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.950486 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.951940 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:28:58.951943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:28:58.951945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:28:58.951967 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:28:58.953297 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.953786 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.953790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.953791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.953881 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:28:58.954195 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:28:58.954198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:28:58.954199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:28:58.954407 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:28:58.954546 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.954790 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.954912 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:28:58.957821 21246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:28:58.961005 21246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:28:58.975888 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.970104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42373 (local address 127.0.0.1:33320)
0619 22:28:58.970662 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:58.970677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:58.970770 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:58.971454 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:58.971458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:58.971476 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:58.971554 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.971562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.972535 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.972541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.974025 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.974036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.975430 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.975434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.975551 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.975578 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:28:58.975671 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:28:58.975743 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":83,"thread_start_us":120,"threads_started":1}
I20260619 22:28:58.975937 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.970268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.5.176.193:42373)
0619 22:28:58.970619 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.970640 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.970872 (+ 232us) server_negotiation.cc:407] Connection header received
0619 22:28:58.970942 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.970946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.971091 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.971248 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:58.971681 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.972410 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.974193 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.975098 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.975207 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.975368 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.975707 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.975803 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":133,"threads_started":1}
I20260619 22:28:58.976051 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.969730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.5.176.195:34813)
0619 22:28:58.970768 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.970781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.970806 (+ 25us) server_negotiation.cc:407] Connection header received
0619 22:28:58.970885 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.970891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.971092 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.971248 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:58.971625 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.972808 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.974468 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.975223 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.975310 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.975418 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.975823 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.975918 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":144,"threads_started":1}
I20260619 22:28:58.975755 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.969385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:40687 (local address 127.0.0.1:49658)
0619 22:28:58.970004 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:58.970057 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:58.970202 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:58.971299 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:58.971301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:58.971313 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:58.971520 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.971527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.972935 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.972938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.974308 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.974316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.975221 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.975224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.975327 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.975363 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 22:28:58.975470 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 22:28:58.975588 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":234,"thread_start_us":88,"threads_started":1}
I20260619 22:28:58.976379 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.970311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.5.176.194:40687)
0619 22:28:58.970790 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:58.970801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:58.970822 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:28:58.970900 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:58.970905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:58.971137 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:58.971460 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:58.971626 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.972781 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.974430 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.975102 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.975222 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.975381 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:58.975823 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 22:28:58.975924 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":94,"threads_started":1}
I20260619 22:28:58.976528 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:58.969587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:34813 (local address 127.0.0.1:53332)
0619 22:28:58.970443 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:58.970458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:58.970546 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:58.971256 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:58.971264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:58.971290 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:58.971481 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.971489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.972956 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.972958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.974323 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:58.974334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:58.975459 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:58.975465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:58.975573 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:58.975593 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 22:28:58.975658 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 22:28:58.975712 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":90,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260619 22:28:58.979211 21371 tablet_service.cc:1467] Processing CreateTablet for tablet 28ccca28b5c04622989bf1789257ea78 (DEFAULT_TABLE table=test-table [id=16da9e2eadf34aa88b2782c17d59bce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:58.979254 21639 tablet_service.cc:1467] Processing CreateTablet for tablet 28ccca28b5c04622989bf1789257ea78 (DEFAULT_TABLE table=test-table [id=16da9e2eadf34aa88b2782c17d59bce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:58.979254 21505 tablet_service.cc:1467] Processing CreateTablet for tablet 28ccca28b5c04622989bf1789257ea78 (DEFAULT_TABLE table=test-table [id=16da9e2eadf34aa88b2782c17d59bce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:28:58.980232 21505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ccca28b5c04622989bf1789257ea78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:58.980216 21371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ccca28b5c04622989bf1789257ea78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:58.980278 21639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ccca28b5c04622989bf1789257ea78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:28:58.986451 21722 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Bootstrap starting.
I20260619 22:28:58.987846 21724 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Bootstrap starting.
I20260619 22:28:58.987874 21723 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Bootstrap starting.
I20260619 22:28:58.989603 21724 tablet_bootstrap.cc:654] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:58.989727 21722 tablet_bootstrap.cc:654] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:58.990211 21723 tablet_bootstrap.cc:654] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Neither blocks nor log segments found. Creating new log.
I20260619 22:28:58.990824 21724 log.cc:824] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:58.990989 21722 log.cc:824] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:58.990988 21723 log.cc:824] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Log is configured to *not* fsync() on all Append() calls
I20260619 22:28:58.992916 21722 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: No bootstrap required, opened a new log
I20260619 22:28:58.992916 21724 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: No bootstrap required, opened a new log
I20260619 22:28:58.993115 21724 ts_tablet_manager.cc:1397] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 22:28:58.993115 21722 ts_tablet_manager.cc:1397] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:28:58.993559 21723 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: No bootstrap required, opened a new log
I20260619 22:28:58.993779 21723 ts_tablet_manager.cc:1397] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:28:58.996598 21723 raft_consensus.cc:348] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:58.996853 21723 raft_consensus.cc:374] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:58.996914 21723 raft_consensus.cc:729] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3203a9cd837e4629ba0ec00fbf4439ee, State: Initialized, Role: FOLLOWER
I20260619 22:28:58.996999 21722 raft_consensus.cc:348] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:58.996999 21724 raft_consensus.cc:348] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:58.997246 21724 raft_consensus.cc:374] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:58.997246 21722 raft_consensus.cc:374] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:28:58.997303 21724 raft_consensus.cc:729] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12711d514b141f9b148429bc80980f7, State: Initialized, Role: FOLLOWER
I20260619 22:28:58.997303 21722 raft_consensus.cc:729] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0623d78435047efb30d1428779d3962, State: Initialized, Role: FOLLOWER
I20260619 22:28:58.997402 21723 consensus_queue.cc:260] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:58.997714 21724 consensus_queue.cc:260] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:58.997714 21722 consensus_queue.cc:260] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:58.998224 21571 heartbeater.cc:503] Master 127.5.176.254:37937 was elected leader, sending a full tablet report...
I20260619 22:28:58.998461 21723 ts_tablet_manager.cc:1428] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 22:28:58.998644 21722 ts_tablet_manager.cc:1428] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:28:58.998644 21724 ts_tablet_manager.cc:1428] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 22:28:58.998972 21437 heartbeater.cc:503] Master 127.5.176.254:37937 was elected leader, sending a full tablet report...
I20260619 22:28:58.999433 21705 heartbeater.cc:503] Master 127.5.176.254:37937 was elected leader, sending a full tablet report...
I20260619 22:28:59.026216 21729 raft_consensus.cc:482] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:28:59.026436 21729 raft_consensus.cc:504] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:59.027616 21729 leader_election.cc:288] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687), e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813)
I20260619 22:28:59.032932 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:59.028049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:43511 (local address 127.5.176.195:34813)
0619 22:28:59.028167 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:59.028172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:59.028473 (+ 301us) server_negotiation.cc:407] Connection header received
0619 22:28:59.028654 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:59.028657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:59.028714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:59.028823 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:59.029054 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.029776 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.031522 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.032165 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.032300 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:59.032354 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:59.032708 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 22:28:59.032792 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:28:59.032932 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:59.028003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:60311 (local address 127.5.176.194:40687)
0619 22:28:59.028170 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:59.028176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:59.028242 (+ 66us) server_negotiation.cc:407] Connection header received
0619 22:28:59.028417 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:59.028420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:59.028470 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:59.028593 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:28:59.028841 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.029640 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.031564 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.032182 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.032300 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:59.032354 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:59.032708 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 22:28:59.032799 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:28:59.032967 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:59.027975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:34813 (local address 127.5.176.193:43511)
0619 22:28:59.028364 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:59.028377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:59.028500 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:59.028841 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:59.028844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:59.028866 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:59.028942 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:59.028946 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.029876 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.029879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:59.031370 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:59.031382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.032303 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.032307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:59.032519 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:59.032579 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 22:28:59.032657 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 22:28:59.032783 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":217,"thread_start_us":146,"threads_started":1}
I20260619 22:28:59.032970 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:59.027805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:40687 (local address 127.5.176.193:60311)
0619 22:28:59.028154 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:59.028170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:28:59.028293 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:28:59.028581 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:28:59.028585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:28:59.028603 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:28:59.028718 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:59.028723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.029789 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.029791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:59.031370 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:28:59.031382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.032303 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.032307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:28:59.032501 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:59.032577 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 22:28:59.032657 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 22:28:59.032780 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":188,"thread_start_us":72,"threads_started":1}
I20260619 22:28:59.033772 21525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccca28b5c04622989bf1789257ea78" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" is_pre_election: true
I20260619 22:28:59.033774 21659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccca28b5c04622989bf1789257ea78" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12711d514b141f9b148429bc80980f7" is_pre_election: true
I20260619 22:28:59.034114 21659 raft_consensus.cc:2454] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 0.
I20260619 22:28:59.034114 21525 raft_consensus.cc:2454] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 0.
I20260619 22:28:59.034683 21327 leader_election.cc:302] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3203a9cd837e4629ba0ec00fbf4439ee, d0623d78435047efb30d1428779d3962, e12711d514b141f9b148429bc80980f7; no voters:
I20260619 22:28:59.035014 21729 raft_consensus.cc:2781] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:28:59.035105 21729 raft_consensus.cc:482] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:28:59.035161 21729 raft_consensus.cc:3037] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:59.036298 21729 raft_consensus.cc:504] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:59.036677 21729 leader_election.cc:288] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 election: Requested vote from peers 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687), e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813)
I20260619 22:28:59.037083 21525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccca28b5c04622989bf1789257ea78" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
I20260619 22:28:59.037083 21659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ccca28b5c04622989bf1789257ea78" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12711d514b141f9b148429bc80980f7"
I20260619 22:28:59.037225 21525 raft_consensus.cc:3037] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:59.037225 21659 raft_consensus.cc:3037] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:28:59.038578 21525 raft_consensus.cc:2454] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 1.
I20260619 22:28:59.038578 21659 raft_consensus.cc:2454] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 1.
I20260619 22:28:59.038951 21326 leader_election.cc:302] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3203a9cd837e4629ba0ec00fbf4439ee, d0623d78435047efb30d1428779d3962, e12711d514b141f9b148429bc80980f7; no voters:
I20260619 22:28:59.039132 21729 raft_consensus.cc:2781] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:28:59.039381 21729 raft_consensus.cc:686] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 LEADER]: Becoming Leader. State: Replica: d0623d78435047efb30d1428779d3962, State: Running, Role: LEADER
I20260619 22:28:59.039747 21729 consensus_queue.cc:237] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:28:59.042804 21244 catalog_manager.cc:5495] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0623d78435047efb30d1428779d3962 (127.5.176.193). New cstate: current_term: 1 leader_uuid: "d0623d78435047efb30d1428779d3962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } health_report { overall_health: HEALTHY } } }
I20260619 22:28:59.055778 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:59.052108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.5.176.193:42373)
0619 22:28:59.052250 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:28:59.052254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:28:59.052268 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:28:59.052398 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:28:59.052401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:28:59.052469 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:28:59.052582 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:28:59.052831 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.053351 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.054255 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:28:59.054489 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:28:59.054604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:28:59.054933 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:28:59.055367 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:28:59.055504 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:28:59.055565 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 22:28:59.055623 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 22:28:59.061864 21659 raft_consensus.cc:1261] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Refusing update from remote peer d0623d78435047efb30d1428779d3962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:59.061864 21525 raft_consensus.cc:1261] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Refusing update from remote peer d0623d78435047efb30d1428779d3962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:28:59.062793 21733 consensus_queue.cc:1035] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:59.063090 21729 consensus_queue.cc:1035] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:28:59.071410 21737 mvcc.cc:205] Tried to move back new op lower bound from 7298695737589448704 to 7298695737509433344. Current Snapshot: MvccSnapshot[applied={T|T < 7298695737589448704}]
I20260619 22:28:59.081526 21246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:59.081728 21246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:28:59.084242 21246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Sending DeleteTablet for 3 replicas of tablet 28ccca28b5c04622989bf1789257ea78
I20260619 22:28:59.085119 21371 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ccca28b5c04622989bf1789257ea78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:59 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:33320
I20260619 22:28:59.085142 21639 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ccca28b5c04622989bf1789257ea78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:59 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:53332
I20260619 22:28:59.085141 21505 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ccca28b5c04622989bf1789257ea78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:59 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:49658
I20260619 22:28:59.085562 21747 tablet_replica.cc:330] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: stopping tablet replica
I20260619 22:28:59.085899 21747 raft_consensus.cc:2229] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:28:59.085952 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21308
I20260619 22:28:59.085934 21749 tablet_replica.cc:330] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: stopping tablet replica
I20260619 22:28:59.086293 21749 raft_consensus.cc:2229] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:59.086743 21749 raft_consensus.cc:2258] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:59.087245 21748 tablet_replica.cc:330] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: stopping tablet replica
I20260619 22:28:59.088789 21749 ts_tablet_manager.cc:1905] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:59.089241 21748 raft_consensus.cc:2229] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:28:59.089555 21748 raft_consensus.cc:2258] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:28:59.090957 21748 ts_tablet_manager.cc:1905] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:28:59.092398 21749 ts_tablet_manager.cc:1918] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:59.092516 21749 log.cc:1196] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/28ccca28b5c04622989bf1789257ea78
I20260619 22:28:59.092850 21749 ts_tablet_manager.cc:1939] T 28ccca28b5c04622989bf1789257ea78 P 3203a9cd837e4629ba0ec00fbf4439ee: Deleting consensus metadata
I20260619 22:28:59.093338 21748 ts_tablet_manager.cc:1918] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:28:59.093425 21748 log.cc:1196] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/28ccca28b5c04622989bf1789257ea78
I20260619 22:28:59.093766 21748 ts_tablet_manager.cc:1939] T 28ccca28b5c04622989bf1789257ea78 P e12711d514b141f9b148429bc80980f7: Deleting consensus metadata
W20260619 22:28:59.093888 21233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:42373 (error 108)
I20260619 22:28:59.094159 21232 catalog_manager.cc:4841] TS 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687): tablet 28ccca28b5c04622989bf1789257ea78 (table test-table [id=16da9e2eadf34aa88b2782c17d59bce3]) successfully deleted
I20260619 22:28:59.094717 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21442
I20260619 22:28:59.094933 21233 catalog_manager.cc:4841] TS e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813): tablet 28ccca28b5c04622989bf1789257ea78 (table test-table [id=16da9e2eadf34aa88b2782c17d59bce3]) successfully deleted
I20260619 22:28:59.095094 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:28:59.094778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42373 (local address 127.0.0.1:33326)
0619 22:28:59.094921 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 22:28:59.094998 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 22:28:59.095561 21233 catalog_manager.cc:4584] TS d0623d78435047efb30d1428779d3962 (127.5.176.193:42373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28ccca28b5c04622989bf1789257ea78: Network error: Client connection negotiation failed: client connection to 127.5.176.193:42373: connect: Connection refused (error 111)
I20260619 22:28:59.101331 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21576
I20260619 22:28:59.107154 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21214
I20260619 22:29:19.114045 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37937
--webserver_interface=127.5.176.254
--webserver_port=36893
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:19.228255 21751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:19.228598 21751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:19.228677 21751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:19.232800 21751 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 22:29:19.232910 21751 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 22:29:19.232939 21751 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 22:29:19.232960 21751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:29:19.233002 21751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:19.233040 21751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:19.233098 21751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:29:19.233139 21751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:29:19.238632 21751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=36893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:19.239805 21751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:19.240888 21751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:19.255494 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.255510 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.255580 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:19.256006 21751 server_base.cc:1034] running on GCE node
I20260619 22:29:19.256784 21751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:19.258248 21751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:19.259428 21751 hybrid_clock.cc:648] HybridClock initialized: now 1781908159259412 us; error 40 us; skew 500 ppm
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:19.262750 21751 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:29:19.263790 21751 webserver.cc:460] Webserver started at http://127.5.176.254:36893/ using document root <none> and password file <none>
I20260619 22:29:19.264451 21751 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:19.264528 21751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:19.269331 21751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:29:19.272277 21767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.273983 21751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 22:29:19.274216 21751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "e293b0e762554081b13d4790e3116e80"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "f714010c13c4601d285430290e2962ab"
server_key_iv: "e600198f11aa2d1b4e1e7378e93e2c79"
server_key_version: "encryptionkey@0"
I20260619 22:29:19.274950 21751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:19.293732 21751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:19.297190 21751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:19.297410 21751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:19.306129 21751 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:37937
I20260619 22:29:19.306612 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:37937 every 8 connection(s)
I20260619 22:29:19.307500 21751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:29:19.312150 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21751
I20260619 22:29:19.312711 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:42373
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:37937
--webserver_port=38383
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:29:19.313012 21820 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:29:19.315515 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap starting.
I20260619 22:29:19.333565 21820 log.cc:824] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:19.342752 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:29:19.343194 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap complete.
I20260619 22:29:19.347760 21820 raft_consensus.cc:348] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:19.348209 21820 raft_consensus.cc:729] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e293b0e762554081b13d4790e3116e80, State: Initialized, Role: FOLLOWER
I20260619 22:29:19.348769 21820 consensus_queue.cc:260] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:19.348928 21820 raft_consensus.cc:388] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:29:19.349015 21820 raft_consensus.cc:482] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:29:19.349152 21820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:29:19.351400 21820 raft_consensus.cc:504] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:19.351775 21820 leader_election.cc:302] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e293b0e762554081b13d4790e3116e80; no voters:
I20260619 22:29:19.352147 21820 leader_election.cc:288] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 22:29:19.352236 21824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:29:19.352458 21824 raft_consensus.cc:686] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 LEADER]: Becoming Leader. State: Replica: e293b0e762554081b13d4790e3116e80, State: Running, Role: LEADER
I20260619 22:29:19.352783 21824 consensus_queue.cc:237] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:19.353566 21820 sys_catalog.cc:564] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:29:19.354614 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e293b0e762554081b13d4790e3116e80. Latest consensus state: current_term: 2 leader_uuid: "e293b0e762554081b13d4790e3116e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } } }
I20260619 22:29:19.354799 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:19.354568 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e293b0e762554081b13d4790e3116e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } } }
I20260619 22:29:19.355000 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:19.355525 21833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:29:19.361647 21833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:29:19.362854 21833 catalog_manager.cc:1259] Loaded cluster ID: c311a325ed1f4585b6d81fb02039c195
I20260619 22:29:19.362921 21833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:29:19.365592 21833 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:29:19.366398 21833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Loaded TSK: 0
I20260619 22:29:19.388614 21833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Generated new TSK 1
I20260619 22:29:19.388999 21833 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:29:19.445441 21822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:19.445781 21822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:19.445858 21822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:19.450204 21822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:19.450297 21822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:19.450394 21822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:29:19.456449 21822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:42373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=38383
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:19.457672 21822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:19.458912 21822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:19.469938 21847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.469938 21850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.469942 21848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:19.470688 21822 server_base.cc:1034] running on GCE node
I20260619 22:29:19.471210 21822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:19.471997 21822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:19.473193 21822 hybrid_clock.cc:648] HybridClock initialized: now 1781908159473157 us; error 64 us; skew 500 ppm
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:19.476325 21822 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:29:19.477420 21822 webserver.cc:460] Webserver started at http://127.5.176.193:38383/ using document root <none> and password file <none>
I20260619 22:29:19.478117 21822 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:19.478266 21822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:19.483083 21822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:19.485805 21857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.487550 21822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 22:29:19.487712 21822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "d0623d78435047efb30d1428779d3962"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "696399bd6972398972dd441ab1d57f22"
server_key_iv: "40fa0a3e36acdf8f9e4f8b9ba1d6f3e7"
server_key_version: "encryptionkey@0"
I20260619 22:29:19.488370 21822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:19.510506 21822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:19.514010 21822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:19.514272 21822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:19.515012 21822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:19.516310 21864 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:29:19.521310 21822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:29:19.521391 21822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260619 22:29:19.521498 21822 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:29:19.523715 21822 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:29:19.523787 21822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 22:29:19.524181 21864 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Bootstrap starting.
I20260619 22:29:19.536226 21822 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:42373
I20260619 22:29:19.536361 21971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:42373 every 8 connection(s)
I20260619 22:29:19.537384 21822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:29:19.540927 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21822
I20260619 22:29:19.541602 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:40687
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:37937
--webserver_port=42665
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:29:19.544134 21864 log.cc:824] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:29:19.553524 21864 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:29:19.554001 21864 tablet_bootstrap.cc:492] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Bootstrap complete.
I20260619 22:29:19.555823 21864 ts_tablet_manager.cc:1397] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.000s
I20260619 22:29:19.559121 21864 raft_consensus.cc:348] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:19.559708 21864 raft_consensus.cc:729] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0623d78435047efb30d1428779d3962, State: Initialized, Role: FOLLOWER
I20260619 22:29:19.560282 21864 consensus_queue.cc:260] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:19.560736 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.540197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:59001 (local address 127.5.176.254:37937)
0619 22:29:19.540772 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.540795 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.541452 (+ 657us) server_negotiation.cc:407] Connection header received
0619 22:29:19.542874 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.542884 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.543314 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.543708 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:19.545447 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.547083 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.549266 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.550221 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.550484 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.554210 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:19.554217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:19.554234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:19.554270 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:19.556667 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.557419 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.557426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.557431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.557578 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.557999 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.558002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.558004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.558496 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:19.558633 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:19.559386 (+ 753us) server_negotiation.cc:299] Negotiation successful
0619 22:29:19.559721 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":158,"threads_started":1}
I20260619 22:29:19.561372 21864 ts_tablet_manager.cc:1428] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 22:29:19.561542 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.540058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:59001)
0619 22:29:19.541278 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:19.541323 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:19.542424 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:19.543900 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:19.543911 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:19.544435 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:19.545198 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.545225 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.547272 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.547277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.549044 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.549059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.550410 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.550415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.550649 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.551456 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:19.551477 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:19.553890 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:19.556813 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.556821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.556840 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.557286 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.557699 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.557703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.557705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.557889 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.558638 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:19.558645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:19.559009 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 22:29:19.559234 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 22:29:19.559608 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":178,"threads_started":1}
I20260619 22:29:19.564693 21972 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37937
I20260619 22:29:19.564973 21972 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:19.565608 21972 heartbeater.cc:511] Master 127.5.176.254:37937 requested a full tablet report, sending...
I20260619 22:29:19.568722 21784 ts_manager.cc:194] Registered new tserver with Master: d0623d78435047efb30d1428779d3962 (127.5.176.193:42373)
I20260619 22:29:19.572907 21784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:59001
I20260619 22:29:19.575897 21972 heartbeater.cc:503] Master 127.5.176.254:37937 was elected leader, sending a full tablet report...
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
I20260619 22:29:19.583822 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.572267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42373 (local address 127.0.0.1:52662)
0619 22:29:19.572883 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:19.572930 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:19.573083 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:19.573681 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:19.573687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:19.574091 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:19.574304 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.574313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.575849 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.575852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.576769 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.576777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.577376 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.577380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.577494 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.578249 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:19.578271 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:19.579963 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:19.581887 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.581898 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.581904 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.582217 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.582542 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.582545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.582547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.582635 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.583116 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:19.583122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:19.583263 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 22:29:19.583407 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 22:29:19.583624 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":150,"threads_started":1}
I20260619 22:29:19.583861 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.572356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.5.176.193:42373)
0619 22:29:19.573266 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.573280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.573299 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:29:19.573356 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.573359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.573525 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.573683 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:19.574468 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.575695 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.576893 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.577251 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.577378 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.580169 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:19.580178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:19.580187 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:19.580209 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:19.581771 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.582352 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.582358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.582359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.582438 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.582734 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.582738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.582739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.582995 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:19.583147 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:19.583505 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 22:29:19.583692 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":108,"threads_started":1}
I20260619 22:29:19.585235 21906 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ccca28b5c04622989bf1789257ea78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:28:59 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52662
I20260619 22:29:19.586360 21981 tablet_replica.cc:330] stopping tablet replica
I20260619 22:29:19.586772 21981 raft_consensus.cc:2229] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:19.587067 21981 raft_consensus.cc:2258] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:19.588354 21981 ts_tablet_manager.cc:1905] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:29:19.593130 21981 ts_tablet_manager.cc:1918] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:29:19.593241 21981 log.cc:1196] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal/wals/28ccca28b5c04622989bf1789257ea78
I20260619 22:29:19.593547 21981 ts_tablet_manager.cc:1939] T 28ccca28b5c04622989bf1789257ea78 P d0623d78435047efb30d1428779d3962: Deleting consensus metadata
I20260619 22:29:19.595712 21771 catalog_manager.cc:4841] TS d0623d78435047efb30d1428779d3962 (127.5.176.193:42373): tablet 28ccca28b5c04622989bf1789257ea78 (table test-table [id=16da9e2eadf34aa88b2782c17d59bce3]) successfully deleted
W20260619 22:29:19.678766 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:19.679108 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:19.679190 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:19.683398 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:19.683470 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:19.683532 21976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:29:19.689085 21976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:40687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=42665
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:19.690300 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:19.691308 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:19.702011 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.702019 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:19.702214 21976 server_base.cc:1034] running on GCE node
W20260619 22:29:19.702018 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:19.703068 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:19.703923 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:19.705112 21976 hybrid_clock.cc:648] HybridClock initialized: now 1781908159705084 us; error 39 us; skew 500 ppm
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:19.708361 21976 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:29:19.709426 21976 webserver.cc:460] Webserver started at http://127.5.176.194:42665/ using document root <none> and password file <none>
I20260619 22:29:19.710212 21976 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:19.710331 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:19.715019 21976 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:19.717558 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.719142 21976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:19.719307 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "14d890684a6728f357f7b1c3d6293697"
server_key_iv: "a690889500d5e9bc47d8164d1cc64276"
server_key_version: "encryptionkey@0"
I20260619 22:29:19.719956 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:19.732453 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:19.735939 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:19.736191 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:19.736838 21976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:19.737776 21976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:19.737847 21976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.737917 21976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:19.737965 21976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.748824 21976 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:40687
I20260619 22:29:19.748948 22109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:40687 every 8 connection(s)
I20260619 22:29:19.750008 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:29:19.750684 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 21976
I20260619 22:29:19.751189 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:34813
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:37937
--webserver_port=34751
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:29:19.767452 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.752484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:60887 (local address 127.5.176.254:37937)
0619 22:29:19.752687 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.752694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.754464 (+ 1770us) server_negotiation.cc:407] Connection header received
0619 22:29:19.755688 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.755694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.755783 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.755899 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:19.757343 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.758450 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.759708 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.760165 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.760309 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.763185 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:19.763192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:19.763198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:19.763232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:19.765178 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.765699 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.765703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.765707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.765796 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.766156 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.766159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.766161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.766400 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:19.766508 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:19.767087 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 22:29:19.767250 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 22:29:19.768267 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.753110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:60887)
0619 22:29:19.754210 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:19.754275 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:19.755334 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:19.756112 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:19.756120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:19.756565 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:19.757163 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.757177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.758608 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.758611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.759557 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.759567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.760363 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.760374 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.760537 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.761271 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:19.761288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:19.762981 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:19.765312 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.765317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.765330 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.765598 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.765891 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.765893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.765895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.766046 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.766514 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:19.766519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:19.766830 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 22:29:19.767022 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 22:29:19.767298 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":196,"threads_started":1}
I20260619 22:29:19.769579 22110 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37937
I20260619 22:29:19.769924 22110 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:19.770713 22110 heartbeater.cc:511] Master 127.5.176.254:37937 requested a full tablet report, sending...
I20260619 22:29:19.772149 21784 ts_manager.cc:194] Registered new tserver with Master: 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687)
I20260619 22:29:19.772784 21784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:60887
W20260619 22:29:19.877800 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:19.878201 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:19.878252 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:19.882511 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:19.882568 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:19.882623 22112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:29:19.888433 22112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:34813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=34751
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:19.889720 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:19.890911 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:19.902109 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.902204 22119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:19.902248 22121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:19.902415 22112 server_base.cc:1034] running on GCE node
I20260619 22:29:19.902961 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:19.903684 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:19.904856 22112 hybrid_clock.cc:648] HybridClock initialized: now 1781908159904840 us; error 41 us; skew 500 ppm
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:19.907979 22112 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:29:19.909008 22112 webserver.cc:460] Webserver started at http://127.5.176.195:34751/ using document root <none> and password file <none>
I20260619 22:29:19.909669 22112 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:19.909770 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:19.914410 22112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:19.916971 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.918583 22112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:19.918696 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "e12711d514b141f9b148429bc80980f7"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "35044919b3be98580443517f45d8e70b"
server_key_iv: "1fe589d6bae55e1d37f47aec51bbc67e"
server_key_version: "encryptionkey@0"
I20260619 22:29:19.919386 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:19.942229 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:19.946484 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:19.946718 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:19.947592 22112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:19.948755 22112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:19.948837 22112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.948908 22112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:19.948941 22112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:19.959426 22112 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:34813
I20260619 22:29:19.959444 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:34813 every 8 connection(s)
I20260619 22:29:19.960641 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:29:19.970530 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22112
Jun 19 22:29:19 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908159, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:29:19.976593 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.962767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:54191 (local address 127.5.176.254:37937)
0619 22:29:19.962936 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.962941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.964438 (+ 1497us) server_negotiation.cc:407] Connection header received
0619 22:29:19.965360 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.965363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.965416 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.965499 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:19.966936 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.967513 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.968722 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.968946 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.969039 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.972324 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:19.972332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:19.972338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:19.972376 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:19.974103 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.974754 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.974760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.974763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.974862 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.975254 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.975258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.975260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.975459 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:19.975573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:19.976255 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 22:29:19.976389 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:29:19.977630 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.963265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:54191)
0619 22:29:19.964254 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:19.964309 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:19.965154 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:19.965676 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:19.965682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:19.966085 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:19.966728 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.966742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.967641 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.967644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.968580 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:19.968589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.969123 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.969126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:19.969280 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.969873 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:19.969895 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:19.972085 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:19.974281 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.974289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.974309 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.974633 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.974970 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:19.974975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:19.974977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:19.975137 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:19.975582 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:19.975591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:19.975934 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 22:29:19.976209 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 22:29:19.976583 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":554,"threads_started":1}
I20260619 22:29:19.979010 22242 heartbeater.cc:347] Connected to a master server at 127.5.176.254:37937
I20260619 22:29:19.979346 22242 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:19.980032 22242 heartbeater.cc:511] Master 127.5.176.254:37937 requested a full tablet report, sending...
I20260619 22:29:19.980756 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.971630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.5.176.254:37937)
0619 22:29:19.971929 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.971937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.972147 (+ 210us) server_negotiation.cc:407] Connection header received
0619 22:29:19.972286 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.972291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.972357 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.972484 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:19.973606 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.974216 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.975144 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.975383 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.975490 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.976998 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:19.977001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:19.977003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:19.977023 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:19.978730 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.979264 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.979267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.979269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.979393 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.979740 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.979743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.979745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.980065 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:19.980274 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:19.980479 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 22:29:19.980600 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
I20260619 22:29:19.981283 21784 ts_manager.cc:194] Registered new tserver with Master: e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813)
I20260619 22:29:19.981946 21784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:54191
I20260619 22:29:19.982882 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:29:19.987361 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.983517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.5.176.254:37937)
0619 22:29:19.983632 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.983635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.984349 (+ 714us) server_negotiation.cc:407] Connection header received
0619 22:29:19.984480 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.984483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.984564 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.984656 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:29:19.984854 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.985403 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.986300 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.986525 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.986608 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.986923 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:29:19.987068 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 22:29:19.987268 (+ 200us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46832: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 22:29:19.987454 22246 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46832: token expired
W20260619 22:29:19.987824 21709 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.176.254:37937)
I20260619 22:29:19.995100 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:19.988216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.5.176.254:37937)
0619 22:29:19.988341 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:19.988344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:19.988357 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:29:19.988389 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:19.988391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:19.988426 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:19.988509 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:19.989092 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.989587 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.990446 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:19.990663 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:19.990749 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:19.991973 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:19.991976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:19.991978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:19.991998 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:19.993605 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.994021 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.994023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.994025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.994096 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:19.994392 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:19.994395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:19.994396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:19.994555 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:19.994665 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:19.994901 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 22:29:19.994988 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:29:19.997934 21783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:29:20.002239 21783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:29:20.012398 21906 tablet_service.cc:1467] Processing CreateTablet for tablet 9a234532aa9d4847a092ea2f7d238987 (DEFAULT_TABLE table=test-table [id=db52609263fc450bbcea7547002ac894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:20.013170 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a234532aa9d4847a092ea2f7d238987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:20.016597 21864 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Bootstrap starting.
I20260619 22:29:20.016942 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.010887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:40687 (local address 127.0.0.1:32948)
0619 22:29:20.011027 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:20.011045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:20.011122 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:20.012357 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:20.012360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:20.012382 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:20.012519 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.012525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.014196 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.014200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.015368 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.015378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.016411 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.016415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.016531 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.016588 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 22:29:20.016689 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 22:29:20.016785 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260619 22:29:20.017031 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.011243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.5.176.194:40687)
0619 22:29:20.011759 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:20.011776 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:20.011803 (+ 27us) server_negotiation.cc:407] Connection header received
0619 22:29:20.011894 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:20.011900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:20.012171 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:20.012705 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:20.012907 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.014025 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.015503 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.016258 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.016373 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.016555 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:20.016737 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 22:29:20.016849 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260619 22:29:20.017645 21864 tablet_bootstrap.cc:654] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:20.019234 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.011609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.5.176.195:34813)
0619 22:29:20.012377 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:20.012399 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:20.012428 (+ 29us) server_negotiation.cc:407] Connection header received
0619 22:29:20.012510 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:20.012514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:20.012704 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:20.012859 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:20.013444 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.014416 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.015916 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.016869 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.016983 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.017418 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:20.018982 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0619 22:29:20.019091 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":284,"threads_started":1}
I20260619 22:29:20.019474 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.011292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:34813 (local address 127.0.0.1:37790)
0619 22:29:20.012021 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:20.012042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:20.012295 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:20.013013 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:20.013015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:20.013031 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:20.013178 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.013185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.014543 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.014547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.015758 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.015770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.018380 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.018385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.018502 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.018526 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 22:29:20.018601 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:29:20.018677 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":130,"threads_started":1}
I20260619 22:29:20.019619 21864 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: No bootstrap required, opened a new log
I20260619 22:29:20.019708 21864 ts_tablet_manager.cc:1397] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:20.020215 21864 raft_consensus.cc:348] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.020334 21864 raft_consensus.cc:374] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:20.020377 21864 raft_consensus.cc:729] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0623d78435047efb30d1428779d3962, State: Initialized, Role: FOLLOWER
I20260619 22:29:20.020510 21864 consensus_queue.cc:260] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.020787 21864 ts_tablet_manager.cc:1428] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 22:29:20.020490 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 9a234532aa9d4847a092ea2f7d238987 (DEFAULT_TABLE table=test-table [id=db52609263fc450bbcea7547002ac894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:20.021687 22044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a234532aa9d4847a092ea2f7d238987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:20.022571 22176 tablet_service.cc:1467] Processing CreateTablet for tablet 9a234532aa9d4847a092ea2f7d238987 (DEFAULT_TABLE table=test-table [id=db52609263fc450bbcea7547002ac894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:20.024008 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a234532aa9d4847a092ea2f7d238987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:20.030514 22255 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Bootstrap starting.
I20260619 22:29:20.031765 22256 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Bootstrap starting.
I20260619 22:29:20.033499 22255 tablet_bootstrap.cc:654] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:20.034516 22255 log.cc:824] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:20.034590 22256 tablet_bootstrap.cc:654] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:20.035301 22256 log.cc:824] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:20.036769 22255 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: No bootstrap required, opened a new log
I20260619 22:29:20.036986 22255 ts_tablet_manager.cc:1397] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 22:29:20.037158 22256 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: No bootstrap required, opened a new log
I20260619 22:29:20.037334 22256 ts_tablet_manager.cc:1397] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260619 22:29:20.038726 21973 tablet.cc:2369] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:29:20.041347 22256 raft_consensus.cc:348] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.041708 22256 raft_consensus.cc:374] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:20.041764 22256 raft_consensus.cc:729] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12711d514b141f9b148429bc80980f7, State: Initialized, Role: FOLLOWER
I20260619 22:29:20.041775 22255 raft_consensus.cc:348] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.042035 22255 raft_consensus.cc:374] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:20.042089 22255 raft_consensus.cc:729] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3203a9cd837e4629ba0ec00fbf4439ee, State: Initialized, Role: FOLLOWER
I20260619 22:29:20.042361 22256 consensus_queue.cc:260] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.042490 22255 consensus_queue.cc:260] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.043190 22110 heartbeater.cc:503] Master 127.5.176.254:37937 was elected leader, sending a full tablet report...
I20260619 22:29:20.043190 22242 heartbeater.cc:503] Master 127.5.176.254:37937 was elected leader, sending a full tablet report...
I20260619 22:29:20.043587 22255 ts_tablet_manager.cc:1428] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:29:20.043587 22256 ts_tablet_manager.cc:1428] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:29:20.140470 21978 raft_consensus.cc:482] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:29:20.140713 21978 raft_consensus.cc:504] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.141880 21978 leader_election.cc:288] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687), e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813)
I20260619 22:29:20.146008 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.142465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47341 (local address 127.5.176.194:40687)
0619 22:29:20.142605 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:20.142608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:20.142723 (+ 115us) server_negotiation.cc:407] Connection header received
0619 22:29:20.142918 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:20.142922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:20.142979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:20.143069 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:20.143335 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.143857 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.144995 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.145411 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.145541 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.145593 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:20.145793 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 22:29:20.145868 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:29:20.146025 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.142305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:40687 (local address 127.5.176.193:47341)
0619 22:29:20.142607 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:20.142624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:20.142752 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:20.143082 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:20.143084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:20.143101 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:20.143220 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.143225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.143977 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.143982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.144872 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.144881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.145538 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.145542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.145636 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.145686 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 22:29:20.145777 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 22:29:20.145892 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":20,"thread_start_us":96,"threads_started":1}
I20260619 22:29:20.146013 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.142438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:53111 (local address 127.5.176.195:34813)
0619 22:29:20.142602 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:20.142607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:20.142949 (+ 342us) server_negotiation.cc:407] Connection header received
0619 22:29:20.143104 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:20.143107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:20.143157 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:20.143239 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:20.143492 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.143982 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.144995 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.145424 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.145563 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.145615 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:20.145806 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 22:29:20.145891 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:29:20.146240 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.142305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:34813 (local address 127.5.176.193:53111)
0619 22:29:20.142848 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:20.142864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:20.142986 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:20.143250 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:20.143252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:20.143268 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:20.143387 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.143393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.144094 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.144099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.144877 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:20.144886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.145549 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.145553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:20.145657 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.145702 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 22:29:20.145792 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 22:29:20.145869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":336,"thread_start_us":91,"threads_started":1}
I20260619 22:29:20.146802 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a234532aa9d4847a092ea2f7d238987" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" is_pre_election: true
I20260619 22:29:20.146978 22196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a234532aa9d4847a092ea2f7d238987" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12711d514b141f9b148429bc80980f7" is_pre_election: true
I20260619 22:29:20.147192 22064 raft_consensus.cc:2454] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 0.
I20260619 22:29:20.147199 22196 raft_consensus.cc:2454] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 0.
I20260619 22:29:20.147670 21861 leader_election.cc:302] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0623d78435047efb30d1428779d3962, e12711d514b141f9b148429bc80980f7; no voters:
I20260619 22:29:20.147960 21978 raft_consensus.cc:2781] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:29:20.148066 21978 raft_consensus.cc:482] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:29:20.148129 21978 raft_consensus.cc:3037] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:20.149151 21978 raft_consensus.cc:504] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.149555 21978 leader_election.cc:288] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 election: Requested vote from peers 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687), e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813)
I20260619 22:29:20.149921 22196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a234532aa9d4847a092ea2f7d238987" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12711d514b141f9b148429bc80980f7"
I20260619 22:29:20.149910 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a234532aa9d4847a092ea2f7d238987" candidate_uuid: "d0623d78435047efb30d1428779d3962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
I20260619 22:29:20.150043 22196 raft_consensus.cc:3037] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:20.150053 22064 raft_consensus.cc:3037] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:20.151383 22196 raft_consensus.cc:2454] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 1.
I20260619 22:29:20.151383 22064 raft_consensus.cc:2454] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0623d78435047efb30d1428779d3962 in term 1.
I20260619 22:29:20.151731 21860 leader_election.cc:302] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3203a9cd837e4629ba0ec00fbf4439ee, d0623d78435047efb30d1428779d3962; no voters:
I20260619 22:29:20.151983 21978 raft_consensus.cc:2781] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:29:20.152284 21978 raft_consensus.cc:686] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 LEADER]: Becoming Leader. State: Replica: d0623d78435047efb30d1428779d3962, State: Running, Role: LEADER
I20260619 22:29:20.152590 21978 consensus_queue.cc:237] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:20.156066 21784 catalog_manager.cc:5495] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0623d78435047efb30d1428779d3962 (127.5.176.193). New cstate: current_term: 1 leader_uuid: "d0623d78435047efb30d1428779d3962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } health_report { overall_health: HEALTHY } } }
I20260619 22:29:20.177848 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.174212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.5.176.193:42373)
0619 22:29:20.174499 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:20.174505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:20.174518 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:29:20.174620 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:20.174622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:20.174676 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:20.174785 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:29:20.175045 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.175575 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.176423 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:20.176666 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:20.176763 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:20.177031 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:29:20.177489 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:29:20.177589 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:20.177650 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 22:29:20.177700 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
I20260619 22:29:20.182431 22064 raft_consensus.cc:1261] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Refusing update from remote peer d0623d78435047efb30d1428779d3962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:29:20.182437 22196 raft_consensus.cc:1261] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Refusing update from remote peer d0623d78435047efb30d1428779d3962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:29:20.183156 21978 consensus_queue.cc:1035] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:29:20.183373 22263 consensus_queue.cc:1035] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:29:20.198287 22267 mvcc.cc:205] Tried to move back new op lower bound from 7298695824100044800 to 7298695823988101120. Current Snapshot: MvccSnapshot[applied={T|T < 7298695824100044800 or (T in {7298695824100044800})}]
I20260619 22:29:20.199816 21784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:29:20.199985 21784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:29:20.202445 21784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Sending DeleteTablet for 3 replicas of tablet 9a234532aa9d4847a092ea2f7d238987
I20260619 22:29:20.203227 21906 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a234532aa9d4847a092ea2f7d238987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:20 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:52662
I20260619 22:29:20.203207 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a234532aa9d4847a092ea2f7d238987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:20 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:32948
I20260619 22:29:20.203508 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21822
I20260619 22:29:20.203621 22276 tablet_replica.cc:330] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: stopping tablet replica
I20260619 22:29:20.203617 22176 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a234532aa9d4847a092ea2f7d238987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:20 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:37790
I20260619 22:29:20.203766 22276 raft_consensus.cc:2229] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:29:20.204444 22278 tablet_replica.cc:330] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: stopping tablet replica
I20260619 22:29:20.204774 22278 raft_consensus.cc:2229] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:20.205084 22278 raft_consensus.cc:2258] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:20.206794 22278 ts_tablet_manager.cc:1905] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:29:20.210337 22278 ts_tablet_manager.cc:1918] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:29:20.210455 22278 log.cc:1196] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/9a234532aa9d4847a092ea2f7d238987
I20260619 22:29:20.210434 22277 tablet_replica.cc:330] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: stopping tablet replica
I20260619 22:29:20.210788 22277 raft_consensus.cc:2229] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:20.210819 22278 ts_tablet_manager.cc:1939] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Deleting consensus metadata
I20260619 22:29:20.211198 22277 raft_consensus.cc:2258] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:20.212204 21771 catalog_manager.cc:4841] TS e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813): tablet 9a234532aa9d4847a092ea2f7d238987 (table test-table [id=db52609263fc450bbcea7547002ac894]) successfully deleted
W20260619 22:29:20.212507 21771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:42373 (error 108)
I20260619 22:29:20.212635 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21976
I20260619 22:29:20.213886 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.213623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42373 (local address 127.0.0.1:52670)
0619 22:29:20.213754 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:20.213818 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 22:29:20.214118 21771 catalog_manager.cc:4584] TS d0623d78435047efb30d1428779d3962 (127.5.176.193:42373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a234532aa9d4847a092ea2f7d238987: Network error: Client connection negotiation failed: client connection to 127.5.176.193:42373: connect: Connection refused (error 111)
I20260619 22:29:20.218679 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 22112
I20260619 22:29:20.219676 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:20.218899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:40687 (local address 127.0.0.1:32956)
0619 22:29:20.219505 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:20.219573 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490}
I20260619 22:29:20.224278 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 21751
I20260619 22:29:40.231523 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37937
--webserver_interface=127.5.176.254
--webserver_port=36893
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:40.353660 22280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:40.353958 22280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:40.354001 22280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:40.358083 22280 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 22:29:40.358175 22280 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 22:29:40.358193 22280 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 22:29:40.358215 22280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:29:40.358235 22280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:40.358312 22280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:40.358335 22280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:29:40.358369 22280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:29:40.363926 22280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=36893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:40.365522 22280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:40.366736 22280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:40.376928 22288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:40.376931 22285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:40.376928 22286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:40.377293 22280 server_base.cc:1034] running on GCE node
I20260619 22:29:40.378106 22280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:40.379482 22280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:40.380692 22280 hybrid_clock.cc:648] HybridClock initialized: now 1781908180380652 us; error 60 us; skew 500 ppm
Jun 19 22:29:40 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908180, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:40.383912 22280 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:29:40.384986 22280 webserver.cc:460] Webserver started at http://127.5.176.254:36893/ using document root <none> and password file <none>
I20260619 22:29:40.385632 22280 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:40.385731 22280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:40.390340 22280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:40.392838 22295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:40.394610 22280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:40.394783 22280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "e293b0e762554081b13d4790e3116e80"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "f714010c13c4601d285430290e2962ab"
server_key_iv: "e600198f11aa2d1b4e1e7378e93e2c79"
server_key_version: "encryptionkey@0"
I20260619 22:29:40.395393 22280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:40.417919 22280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:40.421665 22280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:40.421895 22280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:40.430220 22280 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:37937
I20260619 22:29:40.430634 22347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:37937 every 8 connection(s)
I20260619 22:29:40.431398 22280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:29:40.435973 22348 sys_catalog.cc:263] Verifying existing consensus state
I20260619 22:29:40.437667 22348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap starting.
I20260619 22:29:40.440003 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22280
I20260619 22:29:40.440472 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:42373
--local_ip_for_outbound_sockets=127.5.176.193
--tserver_master_addrs=127.5.176.254:37937
--webserver_port=38383
--webserver_interface=127.5.176.193
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:29:40.455888 22348 log.cc:824] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:40.467283 22348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:29:40.467715 22348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Bootstrap complete.
I20260619 22:29:40.471904 22348 raft_consensus.cc:348] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:40.472345 22348 raft_consensus.cc:729] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e293b0e762554081b13d4790e3116e80, State: Initialized, Role: FOLLOWER
I20260619 22:29:40.472920 22348 consensus_queue.cc:260] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:40.473074 22348 raft_consensus.cc:388] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:29:40.473168 22348 raft_consensus.cc:482] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:29:40.473289 22348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 2 FOLLOWER]: Advancing to term 3
I20260619 22:29:40.475375 22348 raft_consensus.cc:504] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:40.475725 22348 leader_election.cc:302] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e293b0e762554081b13d4790e3116e80; no voters:
I20260619 22:29:40.476120 22348 leader_election.cc:288] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 22:29:40.476212 22352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 22:29:40.476419 22352 raft_consensus.cc:686] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [term 3 LEADER]: Becoming Leader. State: Replica: e293b0e762554081b13d4790e3116e80, State: Running, Role: LEADER
I20260619 22:29:40.476720 22352 consensus_queue.cc:237] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } }
I20260619 22:29:40.477428 22348 sys_catalog.cc:564] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:29:40.477691 22352 sys_catalog.cc:455] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e293b0e762554081b13d4790e3116e80. Latest consensus state: current_term: 3 leader_uuid: "e293b0e762554081b13d4790e3116e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } } }
I20260619 22:29:40.477803 22352 sys_catalog.cc:458] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:40.478199 22356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:29:40.478282 22353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e293b0e762554081b13d4790e3116e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e293b0e762554081b13d4790e3116e80" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 37937 } } }
I20260619 22:29:40.478466 22353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:40.483646 22356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:29:40.484375 22356 catalog_manager.cc:1259] Loaded cluster ID: c311a325ed1f4585b6d81fb02039c195
I20260619 22:29:40.484438 22356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:29:40.488198 22356 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:29:40.488993 22356 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Loaded TSK: 0
I20260619 22:29:40.489077 22356 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e293b0e762554081b13d4790e3116e80: Loaded TSK: 1
F20260619 22:29:40.489938 22356 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd616f359ad google::LogMessage::Fail() at ??:0
@ 0x7fd616f3888a google::LogMessage::SendToLog() at ??:0
@ 0x7fd616f354de google::LogMessage::Flush() at ??:0
@ 0x7fd616f36a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd619bf918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd619bf1f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd62189e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd62187899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd621879178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd6218badf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd6218d982b std::function<>::operator()() at ??:0
@ 0x7fd621878b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd621879f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd621876579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd6218ba640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd6191873d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd619187c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd6191893f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd619178512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd61a7bc6db start_thread at ??:0
@ 0x7fd615a3361f clone at ??:0
W20260619 22:29:40.593043 22350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:40.593329 22350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:40.593374 22350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:40.597569 22350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:40.597633 22350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:40.597720 22350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:29:40.603351 22350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:42373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=38383
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:40.604451 22350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:40.605489 22350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:40.615238 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:40.615252 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:40.615458 22350 server_base.cc:1034] running on GCE node
W20260619 22:29:40.615693 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:40.616225 22350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:40.617081 22350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:40.618225 22350 hybrid_clock.cc:648] HybridClock initialized: now 1781908180618203 us; error 37 us; skew 500 ppm
Jun 19 22:29:40 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908180, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:40.621829 22350 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:29:40.623001 22350 webserver.cc:460] Webserver started at http://127.5.176.193:38383/ using document root <none> and password file <none>
I20260619 22:29:40.623703 22350 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:40.623792 22350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:40.628937 22350 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 22:29:40.631740 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:40.633384 22350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:29:40.633518 22350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "d0623d78435047efb30d1428779d3962"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "696399bd6972398972dd441ab1d57f22"
server_key_iv: "40fa0a3e36acdf8f9e4f8b9ba1d6f3e7"
server_key_version: "encryptionkey@0"
I20260619 22:29:40.634326 22350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:40.656939 22350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:40.660446 22350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:40.660668 22350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:40.661278 22350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:40.662488 22412 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:29:40.667897 22350 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:29:40.667974 22350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 22:29:40.668079 22350 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:29:40.669734 22350 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:29:40.669806 22350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 22:29:40.670095 22412 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Bootstrap starting.
I20260619 22:29:40.680384 22350 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:42373
I20260619 22:29:40.680639 22519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:42373 every 8 connection(s)
I20260619 22:29:40.681403 22350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:29:40.687834 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22350
I20260619 22:29:40.688274 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:40687
--local_ip_for_outbound_sockets=127.5.176.194
--tserver_master_addrs=127.5.176.254:37937
--webserver_port=42665
--webserver_interface=127.5.176.194
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:29:40.692626 22412 log.cc:824] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:40.702486 22412 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:29:40.702960 22412 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Bootstrap complete.
I20260619 22:29:40.704291 22412 ts_tablet_manager.cc:1397] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20260619 22:29:40.707554 22412 raft_consensus.cc:348] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:40.708104 22412 raft_consensus.cc:729] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0623d78435047efb30d1428779d3962, State: Initialized, Role: FOLLOWER
I20260619 22:29:40.708649 22412 consensus_queue.cc:260] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:40.709777 22412 ts_tablet_manager.cc:1428] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260619 22:29:40.731174 22522 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:29:40.683345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:50691)
0619 22:29:40.683820 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.683873 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:40.684759 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:40.730418 (+ 45659us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":142,"threads_started":1}
W20260619 22:29:40.731688 22520 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:37937 (0 consecutive failures): Network error: Failed to ping master at 127.5.176.254:37937: Client connection negotiation failed: client connection to 127.5.176.254:37937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 22:29:40.732473 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.732231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:34563)
0619 22:29:40.732358 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.732419 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 22:29:40.733112 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.732927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:57851)
0619 22:29:40.733025 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.733066 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:29:40.733347 22520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 22:29:40.806195 22523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:40.806466 22523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:40.806548 22523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:40.810631 22523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:40.810693 22523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:40.810793 22523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:29:40.816255 22523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:40687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=42665
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:40.817436 22523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:40.818462 22523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:40.828326 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:40.828326 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:40.828370 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:40.828799 22523 server_base.cc:1034] running on GCE node
I20260619 22:29:40.829198 22523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:40.829860 22523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:40.831071 22523 hybrid_clock.cc:648] HybridClock initialized: now 1781908180831051 us; error 43 us; skew 500 ppm
Jun 19 22:29:40 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908180, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:40.833843 22523 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:29:40.834807 22523 webserver.cc:460] Webserver started at http://127.5.176.194:42665/ using document root <none> and password file <none>
I20260619 22:29:40.835395 22523 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:40.835498 22523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:40.839689 22523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:29:40.841874 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:40.843127 22523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:29:40.843266 22523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "3203a9cd837e4629ba0ec00fbf4439ee"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "14d890684a6728f357f7b1c3d6293697"
server_key_iv: "a690889500d5e9bc47d8164d1cc64276"
server_key_version: "encryptionkey@0"
I20260619 22:29:40.843829 22523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:40.860531 22523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:40.863873 22523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:40.864099 22523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:40.864732 22523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:40.865868 22547 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 22:29:40.870190 22523 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 22:29:40.870301 22523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 22:29:40.870414 22523 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 22:29:40.872201 22523 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 22:29:40.872291 22523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 22:29:40.872531 22547 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Bootstrap starting.
I20260619 22:29:40.878253 22547 log.cc:824] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:40.883810 22523 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:40687
I20260619 22:29:40.883886 22655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:40687 every 8 connection(s)
I20260619 22:29:40.884774 22523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:29:40.886703 22547 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:29:40.887118 22547 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Bootstrap complete.
I20260619 22:29:40.887456 22547 ts_tablet_manager.cc:1397] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260619 22:29:40.887807 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22523
I20260619 22:29:40.888200 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.886753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:34003)
0619 22:29:40.887339 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.887474 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":196,"threads_started":1}
I20260619 22:29:40.888259 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:34813
--local_ip_for_outbound_sockets=127.5.176.195
--tserver_master_addrs=127.5.176.254:37937
--webserver_port=34751
--webserver_interface=127.5.176.195
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:40.888837 22656 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:37937 (0 consecutive failures): Network error: Failed to ping master at 127.5.176.254:37937: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
I20260619 22:29:40.889540 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.889295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:34295)
0619 22:29:40.889418 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.889474 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 22:29:40.890456 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.890183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:51667)
0619 22:29:40.890307 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.890377 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 22:29:40.890646 22656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 22:29:40.891561 22547 raft_consensus.cc:348] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:40.892221 22547 raft_consensus.cc:729] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3203a9cd837e4629ba0ec00fbf4439ee, State: Initialized, Role: FOLLOWER
I20260619 22:29:40.892763 22547 consensus_queue.cc:260] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:40.893923 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.893688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:33029)
0619 22:29:40.893787 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.893839 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 22:29:40.894018 22547 ts_tablet_manager.cc:1428] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:29:40.894675 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:40.894474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:52761)
0619 22:29:40.894574 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:40.894619 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 22:29:40.999563 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:40.999846 22659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:40.999891 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:41.004204 22659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:41.004271 22659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:41.004369 22659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:29:41.009884 22659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:34813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=34751
--tserver_master_addrs=127.5.176.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:41.011039 22659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:41.012079 22659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:41.021729 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:41.021734 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:41.021729 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:41.021822 22659 server_base.cc:1034] running on GCE node
I20260619 22:29:41.022535 22659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:41.023236 22659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:41.024420 22659 hybrid_clock.cc:648] HybridClock initialized: now 1781908181024391 us; error 46 us; skew 500 ppm
Jun 19 22:29:41 dist-test-slave-bt27 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908181, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:41.027246 22659 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:29:41.028211 22659 webserver.cc:460] Webserver started at http://127.5.176.195:34751/ using document root <none> and password file <none>
I20260619 22:29:41.028790 22659 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:41.028882 22659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:41.032737 22659 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:29:41.035179 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:41.036772 22659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:29:41.036912 22659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "e12711d514b141f9b148429bc80980f7"
format_stamp: "Formatted at 2026-06-19 22:28:58 on dist-test-slave-bt27"
server_key: "35044919b3be98580443517f45d8e70b"
server_key_iv: "1fe589d6bae55e1d37f47aec51bbc67e"
server_key_version: "encryptionkey@0"
I20260619 22:29:41.037678 22659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:41.054113 22659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:41.057302 22659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:41.057518 22659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:41.058123 22659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:41.059041 22659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:41.059114 22659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:41.059182 22659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:41.059228 22659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:41.068630 22659 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:34813
I20260619 22:29:41.068655 22788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:34813 every 8 connection(s)
I20260619 22:29:41.069638 22659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:29:41.073341 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:41.071949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:43859)
0619 22:29:41.072581 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:41.072685 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":101,"threads_started":1}
W20260619 22:29:41.073864 22789 heartbeater.cc:644] Failed to heartbeat to 127.5.176.254:37937 (0 consecutive failures): Network error: Failed to ping master at 127.5.176.254:37937: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
I20260619 22:29:41.074532 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:41.074322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:56655)
0619 22:29:41.074436 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:41.074482 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 22:29:41.075086 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:41.074910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:35261)
0619 22:29:41.075001 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:41.075040 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 22:29:41.075263 22789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 22:29:41.076234 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22659
I20260619 22:29:41.734826 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:41.734228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:45605)
0619 22:29:41.734584 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:41.734667 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
I20260619 22:29:41.896023 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:41.895536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:52165)
0619 22:29:41.895851 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:41.895918 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260619 22:29:42.076426 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.075921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:57827)
0619 22:29:42.076230 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.076306 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
I20260619 22:29:42.202780 22796 raft_consensus.cc:482] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:29:42.203051 22796 raft_consensus.cc:504] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:42.204348 22796 leader_election.cc:288] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813), d0623d78435047efb30d1428779d3962 (127.5.176.193:42373)
Jun 19 22:29:42 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908180, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.193@KRBTEST.COM
Jun 19 22:29:42 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908180, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.195@KRBTEST.COM
I20260619 22:29:42.219726 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.205029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:42373 (local address 127.5.176.194:49491)
0619 22:29:42.205448 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.205466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:42.205609 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:42.206822 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:42.206833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:42.207429 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:42.208019 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:42.208035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.209417 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.209419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:42.210478 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:42.210488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.211284 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.211288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:42.211435 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:42.212144 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:42.212162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:42.214041 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:42.216715 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:42.216724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:42.216742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:42.217150 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:42.217607 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:42.217611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:42.217613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:42.217785 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:42.218591 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:42.218599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:42.218897 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 22:29:42.219112 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 22:29:42.219522 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":58,"spinlock_wait_cycles":2944,"thread_start_us":61,"threads_started":1}
I20260619 22:29:42.219739 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.204884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:51797 (local address 127.5.176.195:34813)
0619 22:29:42.205309 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:42.205333 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:42.205384 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:29:42.206516 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:42.206522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:42.206856 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:42.207149 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:42.208191 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.209241 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.210597 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.211102 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.211281 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:42.214722 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:42.214733 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:42.214755 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:42.214794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:42.216942 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:42.217506 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:42.217511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:42.217521 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:42.217695 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:42.218037 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:42.218040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:42.218041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:42.218590 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:42.218734 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:42.219233 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 22:29:42.219498 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
I20260619 22:29:42.219774 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.205519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:49491 (local address 127.5.176.193:42373)
0619 22:29:42.205921 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:42.205927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:42.205953 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:29:42.206192 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:42.206202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:42.206473 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:42.206688 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:42.208212 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.209239 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.210605 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.211151 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.211340 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:42.214335 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:42.214342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:42.214356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:42.214386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:42.216506 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:42.217272 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:42.217282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:42.217288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:42.217483 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:42.217893 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:42.217896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:42.217898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:42.218467 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:42.218627 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:42.219258 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 22:29:42.219547 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260619 22:29:42.220233 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.204692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:34813 (local address 127.5.176.194:51797)
0619 22:29:42.204927 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.204960 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:42.205444 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:42.207098 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:42.207101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:42.207482 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:42.208018 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:42.208034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.209420 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.209423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:42.210474 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:42.210484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:42.211238 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:42.211241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:42.211445 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:42.212316 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:42.212329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:42.214458 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:42.217089 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:42.217091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:42.217093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:42.217384 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:42.217806 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:42.217809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:42.217810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:42.217930 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:42.218718 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:42.218721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:42.218934 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 22:29:42.219121 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 22:29:42.220073 (+ 952us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":655,"spinlock_wait_cycles":60544}
I20260619 22:29:42.221768 22474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a234532aa9d4847a092ea2f7d238987" candidate_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d0623d78435047efb30d1428779d3962" is_pre_election: true
I20260619 22:29:42.222205 22474 raft_consensus.cc:2454] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3203a9cd837e4629ba0ec00fbf4439ee in term 1.
W20260619 22:29:42.223296 22544 leader_election.cc:341] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813): Not found: Tablet not found: 9a234532aa9d4847a092ea2f7d238987
I20260619 22:29:42.223588 22544 leader_election.cc:302] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3203a9cd837e4629ba0ec00fbf4439ee, d0623d78435047efb30d1428779d3962; no voters: e12711d514b141f9b148429bc80980f7
I20260619 22:29:42.223924 22796 raft_consensus.cc:2781] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 22:29:42.224025 22796 raft_consensus.cc:482] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:29:42.224085 22796 raft_consensus.cc:3037] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:29:42.226521 22796 raft_consensus.cc:504] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:42.226933 22796 leader_election.cc:288] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [CANDIDATE]: Term 2 election: Requested vote from peers e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813), d0623d78435047efb30d1428779d3962 (127.5.176.193:42373)
W20260619 22:29:42.227846 22544 leader_election.cc:341] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813): Not found: Tablet not found: 9a234532aa9d4847a092ea2f7d238987
I20260619 22:29:42.227829 22474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a234532aa9d4847a092ea2f7d238987" candidate_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d0623d78435047efb30d1428779d3962"
I20260619 22:29:42.228011 22474 raft_consensus.cc:3037] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 1 FOLLOWER]: Advancing to term 2
I20260619 22:29:42.230520 22474 raft_consensus.cc:2454] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3203a9cd837e4629ba0ec00fbf4439ee in term 2.
I20260619 22:29:42.230901 22544 leader_election.cc:302] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3203a9cd837e4629ba0ec00fbf4439ee, d0623d78435047efb30d1428779d3962; no voters: e12711d514b141f9b148429bc80980f7
I20260619 22:29:42.231109 22796 raft_consensus.cc:2781] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 2 FOLLOWER]: Leader election won for term 2
I20260619 22:29:42.231312 22796 raft_consensus.cc:686] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [term 2 LEADER]: Becoming Leader. State: Replica: 3203a9cd837e4629ba0ec00fbf4439ee, State: Running, Role: LEADER
I20260619 22:29:42.231649 22796 consensus_queue.cc:237] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:42.232262 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.231961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:44655)
0619 22:29:42.232118 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.232181 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 22:29:42.736332 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.735744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:48413)
0619 22:29:42.736134 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.736221 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
W20260619 22:29:42.788060 22544 consensus_peers.cc:487] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee -> Peer e12711d514b141f9b148429bc80980f7 (127.5.176.195:34813): Couldn't send request to peer e12711d514b141f9b148429bc80980f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a234532aa9d4847a092ea2f7d238987. This is attempt 1: this message will repeat every 5th retry.
I20260619 22:29:42.831341 22474 raft_consensus.cc:1261] T 9a234532aa9d4847a092ea2f7d238987 P d0623d78435047efb30d1428779d3962 [term 2 FOLLOWER]: Refusing update from remote peer 3203a9cd837e4629ba0ec00fbf4439ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 22:29:42.832346 22803 consensus_queue.cc:1035] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 22:29:42.834849 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.834521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:49207)
0619 22:29:42.834685 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.834751 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:29:42.838258 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.837776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:36183)
0619 22:29:42.838012 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.838095 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260619 22:29:42.838622 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:42.838296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:47183)
0619 22:29:42.838453 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:42.838524 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 22:29:43.077952 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.077427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:52945)
0619 22:29:43.077749 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.077823 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260619 22:29:43.392041 22811 ts_tablet_manager.cc:927] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Initiating tablet copy from peer 3203a9cd837e4629ba0ec00fbf4439ee (127.5.176.194:40687)
I20260619 22:29:43.392764 22811 tablet_copy_client.cc:316] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.176.194:40687
Jun 19 22:29:43 dist-test-slave-bt27 krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908181, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.194@KRBTEST.COM
I20260619 22:29:43.403695 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.393578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:59461 (local address 127.5.176.194:40687)
0619 22:29:43.393843 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:43.393851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:43.393884 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:29:43.393938 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:43.393943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:43.394125 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:43.394312 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:43.395137 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:43.395875 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:43.397028 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:43.397388 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:43.397479 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:43.400017 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:43.400024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:43.400037 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:43.400060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:43.401756 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:43.402289 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:43.402294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:43.402296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:43.402377 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:43.402647 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:43.402649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:43.402650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:43.402883 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:43.403002 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:43.403433 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 22:29:43.403554 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260619 22:29:43.403717 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.393338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:40687 (local address 127.5.176.195:59461)
0619 22:29:43.393535 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.393561 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:43.393745 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:43.394353 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:43.394360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:43.394856 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:43.395005 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:43.395012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:43.395990 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:43.395993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:43.396898 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:43.396908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:43.397504 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:43.397507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:43.397593 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:43.398225 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:43.398237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:43.399865 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:43.401869 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:43.401877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:43.401883 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:43.402184 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:43.402468 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:43.402471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:43.402472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:43.402557 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:43.403037 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:43.403044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:43.403190 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 22:29:43.403354 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 22:29:43.403524 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 22:29:43.404991 22630 tablet_copy_service.cc:140] P 3203a9cd837e4629ba0ec00fbf4439ee: Received BeginTabletCopySession request for tablet 9a234532aa9d4847a092ea2f7d238987 from peer e12711d514b141f9b148429bc80980f7 ({username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:59461)
I20260619 22:29:43.405146 22630 tablet_copy_service.cc:161] P 3203a9cd837e4629ba0ec00fbf4439ee: Beginning new tablet copy session on tablet 9a234532aa9d4847a092ea2f7d238987 from peer e12711d514b141f9b148429bc80980f7 at {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:59461: session id = e12711d514b141f9b148429bc80980f7-9a234532aa9d4847a092ea2f7d238987
I20260619 22:29:43.406446 22630 tablet_copy_source_session.cc:205] T 9a234532aa9d4847a092ea2f7d238987 P 3203a9cd837e4629ba0ec00fbf4439ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 22:29:43.410772 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a234532aa9d4847a092ea2f7d238987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:43.417284 22811 tablet_copy_client.cc:791] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: tablet copy: Starting download of 0 data blocks...
I20260619 22:29:43.417484 22811 tablet_copy_client.cc:655] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: tablet copy: Starting download of 1 WAL segments...
I20260619 22:29:43.419775 22811 tablet_copy_client.cc:531] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 22:29:43.421509 22811 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Bootstrap starting.
I20260619 22:29:43.427253 22811 log.cc:824] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:43.434352 22811 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 22:29:43.434705 22811 tablet_bootstrap.cc:492] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Bootstrap complete.
I20260619 22:29:43.435047 22811 ts_tablet_manager.cc:1397] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260619 22:29:43.437327 22811 raft_consensus.cc:348] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:43.437574 22811 raft_consensus.cc:729] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12711d514b141f9b148429bc80980f7, State: Initialized, Role: FOLLOWER
I20260619 22:29:43.438031 22811 consensus_queue.cc:260] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3203a9cd837e4629ba0ec00fbf4439ee" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 40687 } } peers { permanent_uuid: "e12711d514b141f9b148429bc80980f7" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 34813 } } peers { permanent_uuid: "d0623d78435047efb30d1428779d3962" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 42373 } }
I20260619 22:29:43.439025 22811 ts_tablet_manager.cc:1428] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 22:29:43.439606 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.439236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:60043)
0619 22:29:43.439399 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.439479 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:29:43.439855 22630 tablet_copy_service.cc:342] P 3203a9cd837e4629ba0ec00fbf4439ee: Request end of tablet copy session e12711d514b141f9b148429bc80980f7-9a234532aa9d4847a092ea2f7d238987 received from {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:59461
I20260619 22:29:43.439983 22630 tablet_copy_service.cc:433] P 3203a9cd837e4629ba0ec00fbf4439ee: ending tablet copy session e12711d514b141f9b148429bc80980f7-9a234532aa9d4847a092ea2f7d238987 on tablet 9a234532aa9d4847a092ea2f7d238987 with peer e12711d514b141f9b148429bc80980f7
I20260619 22:29:43.440553 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.440193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:54983)
0619 22:29:43.440351 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.440428 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 22:29:43.839527 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.839059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:50257)
0619 22:29:43.839332 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.839390 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260619 22:29:43.839737 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.839355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:58539)
0619 22:29:43.839566 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.839639 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260619 22:29:43.960355 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.960098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:48251)
0619 22:29:43.960213 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.960267 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 22:29:43.960505 22743 raft_consensus.cc:1203] T 9a234532aa9d4847a092ea2f7d238987 P e12711d514b141f9b148429bc80980f7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 22:29:43.961582 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:43.961264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:43271)
0619 22:29:43.961471 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:43.961516 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260619 22:29:44.840969 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:44.840521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:34261)
0619 22:29:44.840746 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:44.840875 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260619 22:29:44.961431 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:44.961032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:44223)
0619 22:29:44.961258 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:44.961334 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260619 22:29:44.962592 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:44.962217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:36321)
0619 22:29:44.962466 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:44.962513 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260619 22:29:45.842610 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:45.841971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:50971)
0619 22:29:45.842427 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:45.842507 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":191,"threads_started":1}
I20260619 22:29:45.963196 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:45.962661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:53629)
0619 22:29:45.962981 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:45.963053 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260619 22:29:45.963850 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:45.963284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:50721)
0619 22:29:45.963641 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:45.963717 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260619 22:29:45.964031 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:45.963779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:39119)
0619 22:29:45.963894 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:45.963967 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 22:29:46.844048 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:46.843497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:45101)
0619 22:29:46.843830 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:46.843914 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260619 22:29:46.965433 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:46.964908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:43773)
0619 22:29:46.965244 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:46.965329 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260619 22:29:46.965531 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:46.964998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:46607)
0619 22:29:46.965348 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:46.965415 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":140,"threads_started":1}
I20260619 22:29:47.845289 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:47.844839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:55133)
0619 22:29:47.845114 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:47.845171 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260619 22:29:47.966786 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:47.966388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:35989)
0619 22:29:47.966636 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:47.966692 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260619 22:29:47.966856 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:47.966385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:41349)
0619 22:29:47.966719 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:47.966768 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":90,"threads_started":1}
I20260619 22:29:48.846612 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:48.846123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:45137)
0619 22:29:48.846409 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:48.846506 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260619 22:29:48.968016 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:48.967589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:44015)
0619 22:29:48.967874 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:48.967931 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
I20260619 22:29:48.968025 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:48.967581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:51501)
0619 22:29:48.967866 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:48.967922 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260619 22:29:49.847842 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:49.847388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:46899)
0619 22:29:49.847675 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:49.847737 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
I20260619 22:29:49.969267 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:49.968836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:40963)
0619 22:29:49.969106 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:49.969163 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
I20260619 22:29:49.969358 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:49.968926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:48833)
0619 22:29:49.969215 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:49.969276 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260619 22:29:50.849274 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:50.848786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:54093)
0619 22:29:50.849107 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:50.849165 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260619 22:29:50.970664 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:50.970212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:54747)
0619 22:29:50.970498 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:50.970568 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260619 22:29:50.970705 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:50.970291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:43689)
0619 22:29:50.970567 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:50.970618 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260619 22:29:51.850534 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:51.850066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:42847)
0619 22:29:51.850362 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:51.850420 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":132,"threads_started":1}
I20260619 22:29:51.972150 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:51.971678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:33075)
0619 22:29:51.971989 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:51.972055 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260619 22:29:51.972164 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:51.971678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:47765)
0619 22:29:51.972029 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:51.972082 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":93,"threads_started":1}
I20260619 22:29:52.851763 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:52.851335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:60273)
0619 22:29:52.851599 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:52.851663 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260619 22:29:52.973464 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:52.973019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:57185)
0619 22:29:52.973301 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:52.973359 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
I20260619 22:29:52.973479 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:52.973024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:57993)
0619 22:29:52.973349 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:52.973397 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260619 22:29:53.853127 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:53.852674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:59743)
0619 22:29:53.852969 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:53.853033 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260619 22:29:53.974817 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:53.974373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:41469)
0619 22:29:53.974650 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:53.974712 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260619 22:29:53.974853 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:53.974373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:33835)
0619 22:29:53.974716 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:53.974769 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":104,"threads_started":1}
I20260619 22:29:54.854341 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:54.853908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:48425)
0619 22:29:54.854199 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:54.854249 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260619 22:29:54.976030 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:54.975623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:49783)
0619 22:29:54.975873 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:54.975929 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260619 22:29:54.976075 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:54.975623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:58105)
0619 22:29:54.975937 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:54.975985 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":89,"threads_started":1}
I20260619 22:29:55.855707 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:55.855278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.193:45279)
0619 22:29:55.855554 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:55.855610 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260619 22:29:55.977329 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:55.976867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.195:53117)
0619 22:29:55.977171 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:55.977237 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260619 22:29:55.977367 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:55.976934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:37937 (local address 127.5.176.194:36387)
0619 22:29:55.977234 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:55.977285 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.254:37937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
I20260619 22:29:56.084376 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 22350
I20260619 22:29:56.089979 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 22523
I20260619 22:29:56.095731 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 22659
2026-06-19T22:29:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59334 ms)
[----------] 30 tests from SecurityITest (244568 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:29:56 dist-test-slave-bt27 krb5kdc[22861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:29:56 dist-test-slave-bt27 krb5kdc[22861](info): set up 2 sockets
Jun 19 22:29:56 dist-test-slave-bt27 krb5kdc[22861](info): commencing operation
krb5kdc: starting...
W20260619 22:29:57.150008 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:29:57 dist-test-slave-bt27 krb5kdc[22861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:29:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:29:57.289937 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:32921
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:32921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:57.399699 22874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:57.399994 22874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:57.400036 22874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:57.404089 22874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:29:57.404157 22874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:57.404176 22874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:57.404193 22874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:29:57.404209 22874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:29:57.409585 22874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:32921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:32921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:57.410701 22874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:57.411733 22874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:29:57.420691 22874 server_base.cc:1034] running on GCE node
W20260619 22:29:57.420650 22879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:57.420647 22880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:57.420679 22882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:57.421518 22874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:57.422784 22874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:57.423990 22874 hybrid_clock.cc:648] HybridClock initialized: now 1781908197423969 us; error 36 us; skew 500 ppm
Jun 19 22:29:57 dist-test-slave-bt27 krb5kdc[22861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:57.426926 22874 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:29:57.427842 22874 webserver.cc:460] Webserver started at http://127.5.176.254:39303/ using document root <none> and password file <none>
I20260619 22:29:57.428434 22874 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:57.428525 22874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:57.428758 22874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:57.430987 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "f34accbbddfc40429e807d91a63f75a2"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "b86927e7a60f6ba48c971ee0e8cbe5b8"
server_key_iv: "e2af3ce4ee24fdc10ccd0fc6645dddb8"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.431587 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "f34accbbddfc40429e807d91a63f75a2"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "b86927e7a60f6ba48c971ee0e8cbe5b8"
server_key_iv: "e2af3ce4ee24fdc10ccd0fc6645dddb8"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.435278 22874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 22:29:57.437644 22889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:57.438895 22874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:29:57.439036 22874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "f34accbbddfc40429e807d91a63f75a2"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "b86927e7a60f6ba48c971ee0e8cbe5b8"
server_key_iv: "e2af3ce4ee24fdc10ccd0fc6645dddb8"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.439145 22874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:57.457687 22874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:57.460923 22874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:57.461138 22874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:57.468390 22874 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:32921
I20260619 22:29:57.468410 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:32921 every 8 connection(s)
I20260619 22:29:57.469406 22874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:29:57.472249 22942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:57.476562 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22874
I20260619 22:29:57.476703 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:29:57.476994 5827 external_mini_cluster.cc:1428] Setting key 92430dcd8c25418ea6bd34cac2e1cf92
I20260619 22:29:57.478652 22942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2: Bootstrap starting.
I20260619 22:29:57.480980 22942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:57.481734 22942 log.cc:824] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:57.483640 22942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2: No bootstrap required, opened a new log
Jun 19 22:29:57 dist-test-slave-bt27 krb5kdc[22861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:29:57.488116 22942 raft_consensus.cc:348] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34accbbddfc40429e807d91a63f75a2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 32921 } }
I20260619 22:29:57.488374 22942 raft_consensus.cc:374] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:57.488456 22942 raft_consensus.cc:729] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34accbbddfc40429e807d91a63f75a2, State: Initialized, Role: FOLLOWER
I20260619 22:29:57.489004 22942 consensus_queue.cc:260] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34accbbddfc40429e807d91a63f75a2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 32921 } }
I20260619 22:29:57.489194 22942 raft_consensus.cc:388] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:29:57.489272 22942 raft_consensus.cc:482] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:29:57.489374 22942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:57.490351 22942 raft_consensus.cc:504] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34accbbddfc40429e807d91a63f75a2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 32921 } }
I20260619 22:29:57.490702 22942 leader_election.cc:302] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f34accbbddfc40429e807d91a63f75a2; no voters:
I20260619 22:29:57.491070 22942 leader_election.cc:288] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:29:57.491139 22947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:29:57.491333 22947 raft_consensus.cc:686] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [term 1 LEADER]: Becoming Leader. State: Replica: f34accbbddfc40429e807d91a63f75a2, State: Running, Role: LEADER
I20260619 22:29:57.491693 22947 consensus_queue.cc:237] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34accbbddfc40429e807d91a63f75a2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 32921 } }
I20260619 22:29:57.492046 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:57.478533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.5.176.254:32921)
0619 22:29:57.479025 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:57.479040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:57.479075 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:29:57.479725 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:57.479741 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:57.480076 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:57.480426 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:57.481262 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:57.482355 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:57.483233 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:57.483707 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:57.483877 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:57.486765 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:57.486770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:57.486783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:57.486809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:57.488751 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:57.489295 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:57.489300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:57.489306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:57.489461 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:57.489757 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:57.489761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:57.489763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:57.490249 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:57.490457 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:57.490747 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 22:29:57.491067 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":149,"threads_started":1}
I20260619 22:29:57.492349 22942 sys_catalog.cc:564] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:29:57.493343 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f34accbbddfc40429e807d91a63f75a2. Latest consensus state: current_term: 1 leader_uuid: "f34accbbddfc40429e807d91a63f75a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34accbbddfc40429e807d91a63f75a2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 32921 } } }
I20260619 22:29:57.493501 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:57.494259 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f34accbbddfc40429e807d91a63f75a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34accbbddfc40429e807d91a63f75a2" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 32921 } } }
I20260619 22:29:57.494365 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:57.494645 22956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:29:57.497958 22956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:29:57.503846 22956 catalog_manager.cc:1347] Generated new cluster ID: 72df5d6af38b465d8cd8ef5ed395c605
I20260619 22:29:57.503943 22956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:29:57.519147 22956 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:29:57.520103 22956 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:29:57.527042 22956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2: Generated new TSK 0
I20260619 22:29:57.527752 22956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:29:57.562517 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:32921
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:57.710423 22968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:57.710687 22968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:57.710731 22968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:57.714893 22968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:57.714948 22968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:57.715054 22968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:29:57.720312 22968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:32921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:57.721431 22968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:57.722494 22968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:57.732379 22973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:57.732355 22976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:57.732355 22974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:57.732540 22968 server_base.cc:1034] running on GCE node
I20260619 22:29:57.733067 22968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:57.733739 22968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:57.734900 22968 hybrid_clock.cc:648] HybridClock initialized: now 1781908197734885 us; error 30 us; skew 500 ppm
Jun 19 22:29:57 dist-test-slave-bt27 krb5kdc[22861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:57.737625 22968 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:29:57.738546 22968 webserver.cc:460] Webserver started at http://127.5.176.193:46745/ using document root <none> and password file <none>
I20260619 22:29:57.739114 22968 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:57.739209 22968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:57.739450 22968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:57.741456 22968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "9de4849ad1534bf5ad49a530b3b34ee1"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "264ecb1140fa6400490ff979bdb019d2"
server_key_iv: "25de67e1d272ce0c039f598a9d20be97"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.742097 22968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "9de4849ad1534bf5ad49a530b3b34ee1"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "264ecb1140fa6400490ff979bdb019d2"
server_key_iv: "25de67e1d272ce0c039f598a9d20be97"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.746059 22968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:29:57.748678 22983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:57.750037 22968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:29:57.750242 22968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "9de4849ad1534bf5ad49a530b3b34ee1"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "264ecb1140fa6400490ff979bdb019d2"
server_key_iv: "25de67e1d272ce0c039f598a9d20be97"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.750401 22968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:57.767948 22968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:57.771473 22968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:57.771689 22968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:57.772323 22968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:57.773169 22968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:57.773242 22968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:57.773319 22968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:57.773361 22968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:57.782609 22968 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:34913
I20260619 22:29:57.782626 23096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:34913 every 8 connection(s)
I20260619 22:29:57.783504 22968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:29:57.786455 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 22968
I20260619 22:29:57.786588 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:29:57.786897 5827 external_mini_cluster.cc:1428] Setting key 0c64e13b6ad04e2a6325d353979a33f8
Jun 19 22:29:57 dist-test-slave-bt27 krb5kdc[22861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:29:57.797600 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:57.785274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:57431 (local address 127.5.176.254:32921)
0619 22:29:57.785415 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:57.785419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:57.786189 (+ 770us) server_negotiation.cc:407] Connection header received
0619 22:29:57.787246 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:57.787250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:57.787322 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:57.787413 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:57.788703 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:57.789294 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:57.790515 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:57.790746 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:57.790881 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:57.793722 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:57.793729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:57.793735 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:57.793771 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:57.795515 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:57.796030 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:57.796036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:57.796039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:57.796130 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:57.796474 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:57.796481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:57.796485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:57.796727 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:57.796829 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:57.797364 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 22:29:57.797472 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:29:57.798496 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:57.785585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:32921 (local address 127.5.176.193:57431)
0619 22:29:57.786016 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:57.786058 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:57.787041 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:57.787599 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:57.787607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:57.788073 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:57.788561 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:57.788574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:57.789427 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:57.789430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:57.790392 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:57.790401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:57.790910 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:57.790914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:57.791064 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:57.791687 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:57.791712 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:57.793527 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:57.795642 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:57.795647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:57.795659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:57.795919 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:57.796234 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:57.796236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:57.796237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:57.796372 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:57.796831 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:57.796837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:57.797115 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 22:29:57.797279 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 22:29:57.797545 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260619 22:29:57.799777 23097 heartbeater.cc:347] Connected to a master server at 127.5.176.254:32921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:29:57.800096 23097 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:57.800709 23097 heartbeater.cc:511] Master 127.5.176.254:32921 requested a full tablet report, sending...
I20260619 22:29:57.802412 22906 ts_manager.cc:194] Registered new tserver with Master: 9de4849ad1534bf5ad49a530b3b34ee1 (127.5.176.193:34913)
I20260619 22:29:57.804064 22906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:57431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:29:57.817655 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:32921
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:57.928318 23102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:57.928691 23102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:57.928756 23102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:57.933027 23102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:57.933107 23102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:57.933171 23102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:29:57.938859 23102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:32921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:57.939898 23102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:57.940869 23102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:29:57.950619 23102 server_base.cc:1034] running on GCE node
W20260619 22:29:57.950575 23108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:57.950543 23110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:57.950536 23107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:57.951184 23102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:57.951776 23102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:57.952939 23102 hybrid_clock.cc:648] HybridClock initialized: now 1781908197952908 us; error 45 us; skew 500 ppm
Jun 19 22:29:57 dist-test-slave-bt27 krb5kdc[22861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:57.955861 23102 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:29:57.956763 23102 webserver.cc:460] Webserver started at http://127.5.176.194:43709/ using document root <none> and password file <none>
I20260619 22:29:57.957348 23102 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:57.957412 23102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:57.957636 23102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:57.959815 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "baf1a0560b7b4859b31fa859f6ed1a1d"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "57aa84c9fcac8a33861e5b52512a8792"
server_key_iv: "b512f40ee2510f03d68a7cde06979dd9"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.960390 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "baf1a0560b7b4859b31fa859f6ed1a1d"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "57aa84c9fcac8a33861e5b52512a8792"
server_key_iv: "b512f40ee2510f03d68a7cde06979dd9"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.964284 23102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:29:57.966779 23117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:57.968065 23102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:29:57.968233 23102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "baf1a0560b7b4859b31fa859f6ed1a1d"
format_stamp: "Formatted at 2026-06-19 22:29:57 on dist-test-slave-bt27"
server_key: "57aa84c9fcac8a33861e5b52512a8792"
server_key_iv: "b512f40ee2510f03d68a7cde06979dd9"
server_key_version: "encryptionkey@0"
I20260619 22:29:57.968356 23102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:57.991271 23102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:57.995147 23102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:57.995358 23102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:57.995970 23102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:57.996840 23102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:57.996912 23102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:57.996985 23102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:57.997027 23102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:58.005942 23102 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:43203
I20260619 22:29:58.005965 23230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:43203 every 8 connection(s)
I20260619 22:29:58.006889 23102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:29:58.015064 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23102
I20260619 22:29:58.015172 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:29:58.015456 5827 external_mini_cluster.cc:1428] Setting key 7d80aee3d686a019ac3471787b00adb8
Jun 19 22:29:58 dist-test-slave-bt27 krb5kdc[22861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:29:58.021595 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.008679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:55837 (local address 127.5.176.254:32921)
0619 22:29:58.008811 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.008816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.009709 (+ 893us) server_negotiation.cc:407] Connection header received
0619 22:29:58.010607 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.010610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.010659 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.010733 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:58.011905 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.012459 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.013650 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.013902 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.013998 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.016921 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:58.016924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:58.016927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:58.016948 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:58.018899 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:58.019595 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:58.019601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:58.019604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:58.019720 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:58.020118 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:58.020124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:58.020127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:58.020418 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:58.020544 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.021343 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.021467 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:29:58.022926 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.009044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:32921 (local address 127.5.176.194:55837)
0619 22:29:58.009551 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.009598 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.010413 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.010888 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.010894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.011234 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:58.011703 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.011716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.012601 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.012604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.013511 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.013521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.014048 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.014051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.014309 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.014886 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:58.014904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:58.016706 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:58.019041 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:58.019047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:58.019062 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:58.019479 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:58.019852 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:58.019854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:58.019856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:58.020013 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:58.020555 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:58.020561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:58.021048 (+ 487us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.021275 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.021661 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260619 22:29:58.024209 23231 heartbeater.cc:347] Connected to a master server at 127.5.176.254:32921
I20260619 22:29:58.024487 23231 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:58.025151 23231 heartbeater.cc:511] Master 127.5.176.254:32921 requested a full tablet report, sending...
I20260619 22:29:58.026348 22906 ts_manager.cc:194] Registered new tserver with Master: baf1a0560b7b4859b31fa859f6ed1a1d (127.5.176.194:43203)
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:29:58.026966 22906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:55837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:29:58.047169 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:32921
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:29:58.159037 23236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:58.159315 23236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:58.159360 23236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:58.163502 23236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:58.163570 23236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:58.163632 23236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:29:58.169073 23236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:32921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:58.170197 23236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:58.171206 23236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:58.180963 23241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:58.181047 23242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:58.180982 23244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:58.181370 23236 server_base.cc:1034] running on GCE node
I20260619 22:29:58.181790 23236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:58.182493 23236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:58.183655 23236 hybrid_clock.cc:648] HybridClock initialized: now 1781908198183647 us; error 45 us; skew 500 ppm
Jun 19 22:29:58 dist-test-slave-bt27 krb5kdc[22861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:29:58.186448 23236 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:29:58.187290 23236 webserver.cc:460] Webserver started at http://127.5.176.195:45623/ using document root <none> and password file <none>
I20260619 22:29:58.187824 23236 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:58.187883 23236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:58.188051 23236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:58.190068 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "4a841bd874bb4aac89939925f6893fc2"
format_stamp: "Formatted at 2026-06-19 22:29:58 on dist-test-slave-bt27"
server_key: "6756b8b9a7f69fb505ad75519958b17b"
server_key_iv: "c8217ba1aabb6390eba78084f879068e"
server_key_version: "encryptionkey@0"
I20260619 22:29:58.190663 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "4a841bd874bb4aac89939925f6893fc2"
format_stamp: "Formatted at 2026-06-19 22:29:58 on dist-test-slave-bt27"
server_key: "6756b8b9a7f69fb505ad75519958b17b"
server_key_iv: "c8217ba1aabb6390eba78084f879068e"
server_key_version: "encryptionkey@0"
I20260619 22:29:58.194550 23236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:29:58.196974 23251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:58.198210 23236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:29:58.198310 23236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "4a841bd874bb4aac89939925f6893fc2"
format_stamp: "Formatted at 2026-06-19 22:29:58 on dist-test-slave-bt27"
server_key: "6756b8b9a7f69fb505ad75519958b17b"
server_key_iv: "c8217ba1aabb6390eba78084f879068e"
server_key_version: "encryptionkey@0"
I20260619 22:29:58.198398 23236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:58.213747 23236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:58.217041 23236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:58.217190 23236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:58.217707 23236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:58.218504 23236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:58.218550 23236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:58.218591 23236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:58.218616 23236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:58.227855 23236 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:40935
I20260619 22:29:58.227943 23364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:40935 every 8 connection(s)
I20260619 22:29:58.228785 23236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:29:58.233668 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23236
I20260619 22:29:58.233821 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:29:58.234179 5827 external_mini_cluster.cc:1428] Setting key 4d7c92938ddcb59f2f875f7bb3729b51
Jun 19 22:29:58 dist-test-slave-bt27 krb5kdc[22861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:29:58.241681 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.230639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:56323 (local address 127.5.176.254:32921)
0619 22:29:58.230793 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.230797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.232011 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 22:29:58.232808 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.232810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.232862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.232954 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:58.234188 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.234845 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.236202 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.236440 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.236523 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.238807 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:58.238810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:58.238812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:58.238834 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:58.240038 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:58.240467 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:58.240469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:58.240470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:58.240539 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:58.240811 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:58.240813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:58.240814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:58.240974 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:58.241046 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.241468 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.241561 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:29:58.242609 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.230998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:32921 (local address 127.5.176.195:56323)
0619 22:29:58.231874 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.231913 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.232637 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.233108 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.233114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.233454 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:58.234013 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.234028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.235001 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.235008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.236077 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.236088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.236566 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.236569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.236708 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.237225 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:29:58.237240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:58.238665 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:58.240137 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:58.240141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:58.240152 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:58.240383 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:58.240616 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:29:58.240618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:29:58.240619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:29:58.240738 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:29:58.241057 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:58.241061 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:58.241275 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.241428 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.241660 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":132,"threads_started":1}
I20260619 22:29:58.243813 23365 heartbeater.cc:347] Connected to a master server at 127.5.176.254:32921
I20260619 22:29:58.244053 23365 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:58.244560 23365 heartbeater.cc:511] Master 127.5.176.254:32921 requested a full tablet report, sending...
I20260619 22:29:58.245558 22906 ts_manager.cc:194] Registered new tserver with Master: 4a841bd874bb4aac89939925f6893fc2 (127.5.176.195:40935)
I20260619 22:29:58.246204 22906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:56323
I20260619 22:29:58.247835 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:29:58.257035 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.249805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.5.176.254:32921)
0619 22:29:58.249929 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.249940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.250096 (+ 156us) server_negotiation.cc:407] Connection header received
0619 22:29:58.250241 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.250250 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.250287 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.250352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:58.251052 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.251613 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.252437 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.252659 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.252742 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.254073 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:58.254076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:58.254078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:29:58.254099 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:58.255695 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:58.256083 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:58.256085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:58.256086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:58.256157 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:29:58.256369 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:29:58.256371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:29:58.256372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:29:58.256568 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:58.256669 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.256817 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.256919 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 22:29:58.259738 22906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:29:58.262720 22906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:29:58.276825 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.271662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34913 (local address 127.0.0.1:57872)
0619 22:29:58.272360 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.272374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.272448 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.272982 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.272985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.273004 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:58.273091 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.273097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.274502 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.274509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.275448 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.275458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.276251 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.276255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.276356 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.276393 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.276534 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.276651 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":5,"thread_start_us":78,"threads_started":1}
I20260619 22:29:58.276993 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.271072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.5.176.194:43203)
0619 22:29:58.272381 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.272393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.272413 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:29:58.272485 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.272490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.272653 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.272805 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:58.273033 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.273796 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.275621 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.276429 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.276537 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.276660 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.276758 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.276832 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":133,"threads_started":1}
I20260619 22:29:58.277151 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.272056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.5.176.193:34913)
0619 22:29:58.272452 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.272470 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.272488 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:29:58.272551 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.272555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.272696 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.272820 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:58.273218 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.274346 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.275575 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.276128 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.276251 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.276411 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.276902 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.277017 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":117,"threads_started":1}
I20260619 22:29:58.277009 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.270951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43203 (local address 127.0.0.1:36658)
0619 22:29:58.271986 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.272000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.272089 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.272836 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.272839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.272855 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:58.272948 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.272954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.273930 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.273936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.275477 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.275489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.276546 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.276549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.276629 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.276649 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.276729 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.276785 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":89,"thread_start_us":105,"threads_started":1}
I20260619 22:29:58.277774 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.270945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40935 (local address 127.0.0.1:58478)
0619 22:29:58.271587 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.271639 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.271768 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.272521 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.272526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.272550 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:58.272663 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.272667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.274209 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.274212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.275494 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.275500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.277314 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.277318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.277434 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.277480 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.277555 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.277640 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":104,"thread_start_us":81,"threads_started":1}
I20260619 22:29:58.277860 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.271317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.5.176.195:40935)
0619 22:29:58.271740 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.271753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.271779 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:29:58.271972 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.271978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.272188 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.272340 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:58.272897 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.274050 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.275621 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.276424 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.276537 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.276660 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.277603 (+ 943us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.277711 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260619 22:29:58.280479 23299 tablet_service.cc:1467] Processing CreateTablet for tablet 472b8cc31b7a4186b6fcd1208195f572 (DEFAULT_TABLE table=test-table [id=be530b4e771f44fb80f054da09e4881e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:58.280521 23031 tablet_service.cc:1467] Processing CreateTablet for tablet 472b8cc31b7a4186b6fcd1208195f572 (DEFAULT_TABLE table=test-table [id=be530b4e771f44fb80f054da09e4881e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:58.280521 23165 tablet_service.cc:1467] Processing CreateTablet for tablet 472b8cc31b7a4186b6fcd1208195f572 (DEFAULT_TABLE table=test-table [id=be530b4e771f44fb80f054da09e4881e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:58.281435 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472b8cc31b7a4186b6fcd1208195f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:58.281497 23031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472b8cc31b7a4186b6fcd1208195f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:58.281520 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472b8cc31b7a4186b6fcd1208195f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:58.288079 23382 tablet_bootstrap.cc:492] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Bootstrap starting.
I20260619 22:29:58.289054 23383 tablet_bootstrap.cc:492] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Bootstrap starting.
I20260619 22:29:58.290455 23382 tablet_bootstrap.cc:654] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:58.290473 23384 tablet_bootstrap.cc:492] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: Bootstrap starting.
I20260619 22:29:58.290941 23383 tablet_bootstrap.cc:654] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:58.291548 23382 log.cc:824] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:58.292115 23383 log.cc:824] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:58.292737 23384 tablet_bootstrap.cc:654] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:58.293980 23384 log.cc:824] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:58.294170 23383 tablet_bootstrap.cc:492] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: No bootstrap required, opened a new log
I20260619 22:29:58.294183 23382 tablet_bootstrap.cc:492] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: No bootstrap required, opened a new log
I20260619 22:29:58.294368 23383 ts_tablet_manager.cc:1397] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:29:58.294396 23382 ts_tablet_manager.cc:1397] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:29:58.295903 23384 tablet_bootstrap.cc:492] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: No bootstrap required, opened a new log
I20260619 22:29:58.296087 23384 ts_tablet_manager.cc:1397] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:29:58.297415 23382 raft_consensus.cc:348] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.297663 23382 raft_consensus.cc:374] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:58.297724 23382 raft_consensus.cc:729] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a841bd874bb4aac89939925f6893fc2, State: Initialized, Role: FOLLOWER
I20260619 22:29:58.298226 23382 consensus_queue.cc:260] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.298257 23383 raft_consensus.cc:348] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.298542 23383 raft_consensus.cc:374] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:58.298601 23383 raft_consensus.cc:729] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf1a0560b7b4859b31fa859f6ed1a1d, State: Initialized, Role: FOLLOWER
I20260619 22:29:58.298993 23365 heartbeater.cc:503] Master 127.5.176.254:32921 was elected leader, sending a full tablet report...
I20260619 22:29:58.299055 23383 consensus_queue.cc:260] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.299743 23382 ts_tablet_manager.cc:1428] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:29:58.299952 23383 ts_tablet_manager.cc:1428] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:29:58.300011 23384 raft_consensus.cc:348] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.300248 23384 raft_consensus.cc:374] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:58.300308 23384 raft_consensus.cc:729] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de4849ad1534bf5ad49a530b3b34ee1, State: Initialized, Role: FOLLOWER
I20260619 22:29:58.300297 23231 heartbeater.cc:503] Master 127.5.176.254:32921 was elected leader, sending a full tablet report...
I20260619 22:29:58.300724 23384 consensus_queue.cc:260] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.301254 23097 heartbeater.cc:503] Master 127.5.176.254:32921 was elected leader, sending a full tablet report...
I20260619 22:29:58.301570 23384 ts_tablet_manager.cc:1428] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 22:29:58.479748 23366 tablet.cc:2369] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:29:58.508030 23232 tablet.cc:2369] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:29:58.512769 23390 raft_consensus.cc:482] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:29:58.513019 23390 raft_consensus.cc:504] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.514312 23390 leader_election.cc:288] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf1a0560b7b4859b31fa859f6ed1a1d (127.5.176.194:43203), 4a841bd874bb4aac89939925f6893fc2 (127.5.176.195:40935)
I20260619 22:29:58.518607 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.514655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:60281 (local address 127.5.176.194:43203)
0619 22:29:58.514805 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.514810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.515096 (+ 286us) server_negotiation.cc:407] Connection header received
0619 22:29:58.515234 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.515239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.515305 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.515390 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:58.515758 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.516287 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.517585 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.517980 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.518111 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.518184 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.518411 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.518485 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:29:58.518625 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.514568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:40935 (local address 127.5.176.193:39859)
0619 22:29:58.515091 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.515107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.515220 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.515525 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.515527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.515540 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:58.515670 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.515675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.516524 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.516526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.517380 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.517390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.518175 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.518179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.518277 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.518294 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.518381 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.518458 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":64,"thread_start_us":105,"threads_started":1}
I20260619 22:29:58.518644 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.514713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:39859 (local address 127.5.176.195:40935)
0619 22:29:58.514884 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.514889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.515214 (+ 325us) server_negotiation.cc:407] Connection header received
0619 22:29:58.515326 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.515329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.515408 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.515491 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:58.515846 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.516411 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.517510 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.518000 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.518172 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.518230 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.518409 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.518532 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 22:29:58.518643 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.514561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:43203 (local address 127.5.176.193:60281)
0619 22:29:58.514988 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.515001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:58.515111 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:58.515413 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:58.515416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:58.515437 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:58.515632 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.515639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.516405 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.516409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.517416 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:58.517426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.518118 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.518122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:58.518253 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.518290 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 22:29:58.518389 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 22:29:58.518472 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":66,"thread_start_us":135,"threads_started":1}
I20260619 22:29:58.519495 23185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b8cc31b7a4186b6fcd1208195f572" candidate_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" is_pre_election: true
I20260619 22:29:58.519479 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b8cc31b7a4186b6fcd1208195f572" candidate_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a841bd874bb4aac89939925f6893fc2" is_pre_election: true
I20260619 22:29:58.519804 23185 raft_consensus.cc:2454] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de4849ad1534bf5ad49a530b3b34ee1 in term 0.
I20260619 22:29:58.519804 23319 raft_consensus.cc:2454] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de4849ad1534bf5ad49a530b3b34ee1 in term 0.
I20260619 22:29:58.520335 22985 leader_election.cc:302] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a841bd874bb4aac89939925f6893fc2, 9de4849ad1534bf5ad49a530b3b34ee1, baf1a0560b7b4859b31fa859f6ed1a1d; no voters:
I20260619 22:29:58.520646 23390 raft_consensus.cc:2781] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:29:58.520754 23390 raft_consensus.cc:482] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:29:58.520851 23390 raft_consensus.cc:3037] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:58.522095 23390 raft_consensus.cc:504] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.522603 23390 leader_election.cc:288] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [CANDIDATE]: Term 1 election: Requested vote from peers baf1a0560b7b4859b31fa859f6ed1a1d (127.5.176.194:43203), 4a841bd874bb4aac89939925f6893fc2 (127.5.176.195:40935)
I20260619 22:29:58.523013 23185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b8cc31b7a4186b6fcd1208195f572" candidate_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d"
I20260619 22:29:58.523108 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b8cc31b7a4186b6fcd1208195f572" candidate_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a841bd874bb4aac89939925f6893fc2"
I20260619 22:29:58.523180 23185 raft_consensus.cc:3037] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:58.523313 23319 raft_consensus.cc:3037] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:58.524397 23185 raft_consensus.cc:2454] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de4849ad1534bf5ad49a530b3b34ee1 in term 1.
I20260619 22:29:58.524411 23319 raft_consensus.cc:2454] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de4849ad1534bf5ad49a530b3b34ee1 in term 1.
I20260619 22:29:58.524834 22985 leader_election.cc:302] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a841bd874bb4aac89939925f6893fc2, 9de4849ad1534bf5ad49a530b3b34ee1; no voters:
I20260619 22:29:58.525045 23390 raft_consensus.cc:2781] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:29:58.525285 23390 raft_consensus.cc:686] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 1 LEADER]: Becoming Leader. State: Replica: 9de4849ad1534bf5ad49a530b3b34ee1, State: Running, Role: LEADER
I20260619 22:29:58.525621 23390 consensus_queue.cc:237] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } }
I20260619 22:29:58.528734 22904 catalog_manager.cc:5495] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9de4849ad1534bf5ad49a530b3b34ee1 (127.5.176.193). New cstate: current_term: 1 leader_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9de4849ad1534bf5ad49a530b3b34ee1" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34913 } health_report { overall_health: HEALTHY } } }
W20260619 22:29:58.534829 23098 tablet.cc:2369] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:29:58.570639 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.567147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.5.176.193:34913)
0619 22:29:58.567291 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.567295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.567315 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:29:58.567390 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.567392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.567446 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.567546 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:29:58.567802 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.568306 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.569124 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.569333 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.569418 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.569714 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:29:58.570211 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:29:58.570306 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.570396 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.570447 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 22:29:58.576246 23185 raft_consensus.cc:1261] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 1 FOLLOWER]: Refusing update from remote peer 9de4849ad1534bf5ad49a530b3b34ee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:29:58.576246 23319 raft_consensus.cc:1261] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 1 FOLLOWER]: Refusing update from remote peer 9de4849ad1534bf5ad49a530b3b34ee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:29:58.576939 23393 consensus_queue.cc:1035] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf1a0560b7b4859b31fa859f6ed1a1d" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:29:58.577169 23390 consensus_queue.cc:1035] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a841bd874bb4aac89939925f6893fc2" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:29:58.584342 23396 mvcc.cc:205] Tried to move back new op lower bound from 7298695981361451008 to 7298695981164011520. Current Snapshot: MvccSnapshot[applied={T|T < 7298695981361451008}]
I20260619 22:29:58.586125 23398 mvcc.cc:205] Tried to move back new op lower bound from 7298695981361451008 to 7298695981164011520. Current Snapshot: MvccSnapshot[applied={T|T < 7298695981361451008}]
I20260619 22:29:58.586329 23397 mvcc.cc:205] Tried to move back new op lower bound from 7298695981361451008 to 7298695981164011520. Current Snapshot: MvccSnapshot[applied={T|T < 7298695981361451008}]
I20260619 22:29:58.592885 22906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:29:58.593084 22906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:29:58.595577 22906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f34accbbddfc40429e807d91a63f75a2: Sending DeleteTablet for 3 replicas of tablet 472b8cc31b7a4186b6fcd1208195f572
I20260619 22:29:58.596437 23299 tablet_service.cc:1514] Processing DeleteTablet for tablet 472b8cc31b7a4186b6fcd1208195f572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:58 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:58478
I20260619 22:29:58.596539 23165 tablet_service.cc:1514] Processing DeleteTablet for tablet 472b8cc31b7a4186b6fcd1208195f572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:58 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:36658
I20260619 22:29:58.596601 23031 tablet_service.cc:1514] Processing DeleteTablet for tablet 472b8cc31b7a4186b6fcd1208195f572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:58 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:57872
I20260619 22:29:58.597002 23408 tablet_replica.cc:330] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1: stopping tablet replica
I20260619 22:29:58.597338 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 22968
I20260619 22:29:58.597316 23408 raft_consensus.cc:2229] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:29:58.597724 23408 raft_consensus.cc:2258] T 472b8cc31b7a4186b6fcd1208195f572 P 9de4849ad1534bf5ad49a530b3b34ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:58.597990 23407 tablet_replica.cc:330] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: stopping tablet replica
I20260619 22:29:58.598312 23407 raft_consensus.cc:2229] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:58.598616 23407 raft_consensus.cc:2258] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:58.598966 23406 tablet_replica.cc:330] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: stopping tablet replica
I20260619 22:29:58.599246 23406 raft_consensus.cc:2229] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:58.599527 23406 raft_consensus.cc:2258] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:58.600313 23407 ts_tablet_manager.cc:1905] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:29:58.600811 23406 ts_tablet_manager.cc:1905] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:29:58.602707 23407 ts_tablet_manager.cc:1918] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:29:58.602814 23407 log.cc:1196] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/472b8cc31b7a4186b6fcd1208195f572
I20260619 22:29:58.603101 23407 ts_tablet_manager.cc:1939] T 472b8cc31b7a4186b6fcd1208195f572 P baf1a0560b7b4859b31fa859f6ed1a1d: Deleting consensus metadata
I20260619 22:29:58.604125 23406 ts_tablet_manager.cc:1918] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:29:58.604225 23406 log.cc:1196] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/472b8cc31b7a4186b6fcd1208195f572
I20260619 22:29:58.604250 22892 catalog_manager.cc:4841] TS baf1a0560b7b4859b31fa859f6ed1a1d (127.5.176.194:43203): tablet 472b8cc31b7a4186b6fcd1208195f572 (table test-table [id=be530b4e771f44fb80f054da09e4881e]) successfully deleted
I20260619 22:29:58.604578 23406 ts_tablet_manager.cc:1939] T 472b8cc31b7a4186b6fcd1208195f572 P 4a841bd874bb4aac89939925f6893fc2: Deleting consensus metadata
I20260619 22:29:58.605767 22891 catalog_manager.cc:4841] TS 4a841bd874bb4aac89939925f6893fc2 (127.5.176.195:40935): tablet 472b8cc31b7a4186b6fcd1208195f572 (table test-table [id=be530b4e771f44fb80f054da09e4881e]) successfully deleted
W20260619 22:29:58.606496 22891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:34913 (error 108)
I20260619 22:29:58.606738 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23102
I20260619 22:29:58.608011 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.607640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34913 (local address 127.0.0.1:34004)
0619 22:29:58.607832 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:58.607905 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:34913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 22:29:58.608186 22891 catalog_manager.cc:4584] TS 9de4849ad1534bf5ad49a530b3b34ee1 (127.5.176.193:34913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 472b8cc31b7a4186b6fcd1208195f572: Network error: Client connection negotiation failed: client connection to 127.5.176.193:34913: connect: Connection refused (error 111)
I20260619 22:29:58.612857 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23236
I20260619 22:29:58.618335 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 22874
2026-06-19T22:29:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2521 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T22:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:29:58Z Disabled control of system clock
I20260619 22:29:58.648860 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:36785
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:29:58.761672 23413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:58.761979 23413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:58.762022 23413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:58.766183 23413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:29:58.766263 23413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:58.766283 23413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:58.766302 23413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:29:58.766320 23413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:29:58.771308 23413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:36785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:58.772437 23413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:58.773821 23413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:58.782627 23419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:58.782627 23418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:58.782676 23421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:58.782894 23413 server_base.cc:1034] running on GCE node
I20260619 22:29:58.783526 23413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:58.784777 23413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:58.785993 23413 hybrid_clock.cc:648] HybridClock initialized: now 1781908198785979 us; error 33 us; skew 500 ppm
I20260619 22:29:58.788021 23413 webserver.cc:460] Webserver started at http://127.5.176.254:36103/ using document root <none> and password file <none>
I20260619 22:29:58.788611 23413 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:58.788702 23413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:58.788937 23413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:58.791059 23413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "6997a8b3391f4e408e5d0260dec73b28"
format_stamp: "Formatted at 2026-06-19 22:29:58 on dist-test-slave-bt27"
server_key: "696d7211de3447fc45f297b15d0834cf"
server_key_iv: "6e5a640ca051d4e530f27864a7e06ff7"
server_key_version: "encryptionkey@0"
I20260619 22:29:58.791617 23413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "6997a8b3391f4e408e5d0260dec73b28"
format_stamp: "Formatted at 2026-06-19 22:29:58 on dist-test-slave-bt27"
server_key: "696d7211de3447fc45f297b15d0834cf"
server_key_iv: "6e5a640ca051d4e530f27864a7e06ff7"
server_key_version: "encryptionkey@0"
I20260619 22:29:58.795250 23413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 22:29:58.797590 23427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:58.798815 23413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 22:29:58.798951 23413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "6997a8b3391f4e408e5d0260dec73b28"
format_stamp: "Formatted at 2026-06-19 22:29:58 on dist-test-slave-bt27"
server_key: "696d7211de3447fc45f297b15d0834cf"
server_key_iv: "6e5a640ca051d4e530f27864a7e06ff7"
server_key_version: "encryptionkey@0"
I20260619 22:29:58.799062 23413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:58.808949 23413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:58.809585 23413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:58.809768 23413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:58.816748 23413 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:36785
I20260619 22:29:58.816756 23479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:36785 every 8 connection(s)
I20260619 22:29:58.817675 23413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:29:58.820830 23480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:58.825022 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23413
I20260619 22:29:58.825136 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:29:58.825433 5827 external_mini_cluster.cc:1428] Setting key 4347583bf41e6dd66fd8bd9b77221ee5
I20260619 22:29:58.826969 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28: Bootstrap starting.
I20260619 22:29:58.829770 23480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:58.830698 23480 log.cc:824] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:58.832788 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28: No bootstrap required, opened a new log
I20260619 22:29:58.836198 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:58.827016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.5.176.254:36785)
0619 22:29:58.827750 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:58.827771 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:58.827813 (+ 42us) server_negotiation.cc:407] Connection header received
0619 22:29:58.828707 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:58.828728 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:58.829218 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:58.829586 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:58.830078 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.831464 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.832384 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:58.833065 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:58.833232 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:58.834080 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:58.834086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:58.834102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:29:58.834640 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:58.834707 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:29:58.834733 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:58.834858 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:58.834973 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 22:29:58.835249 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":126,"threads_started":1}
I20260619 22:29:58.837446 23480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997a8b3391f4e408e5d0260dec73b28" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36785 } }
I20260619 22:29:58.837664 23480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:58.837739 23480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6997a8b3391f4e408e5d0260dec73b28, State: Initialized, Role: FOLLOWER
I20260619 22:29:58.838189 23480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997a8b3391f4e408e5d0260dec73b28" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36785 } }
I20260619 22:29:58.838374 23480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:29:58.838443 23480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:29:58.838547 23480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:58.839414 23480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997a8b3391f4e408e5d0260dec73b28" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36785 } }
I20260619 22:29:58.839741 23480 leader_election.cc:302] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6997a8b3391f4e408e5d0260dec73b28; no voters:
I20260619 22:29:58.840016 23480 leader_election.cc:288] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:29:58.840138 23485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:29:58.840363 23485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [term 1 LEADER]: Becoming Leader. State: Replica: 6997a8b3391f4e408e5d0260dec73b28, State: Running, Role: LEADER
I20260619 22:29:58.840662 23485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997a8b3391f4e408e5d0260dec73b28" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36785 } }
I20260619 22:29:58.841198 23480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:29:58.842238 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6997a8b3391f4e408e5d0260dec73b28. Latest consensus state: current_term: 1 leader_uuid: "6997a8b3391f4e408e5d0260dec73b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997a8b3391f4e408e5d0260dec73b28" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36785 } } }
I20260619 22:29:58.842373 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:58.842654 23486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6997a8b3391f4e408e5d0260dec73b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997a8b3391f4e408e5d0260dec73b28" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36785 } } }
I20260619 22:29:58.842763 23486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28 [sys.catalog]: This master's current role is: LEADER
I20260619 22:29:58.843637 23494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:29:58.846403 23494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:29:58.852108 23494 catalog_manager.cc:1347] Generated new cluster ID: 9975f47bfa504df9a521eae433e79c21
I20260619 22:29:58.852186 23494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:29:58.879972 23494 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:29:58.881250 23494 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:29:58.893143 23494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28: Generated new TSK 0
I20260619 22:29:58.893985 23494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:29:58.906422 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36785
--builtin_ntp_servers=127.5.176.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:29:59.022941 23504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:59.023234 23504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:59.023275 23504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:59.027386 23504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:59.027446 23504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:59.027503 23504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:29:59.032294 23504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:59.033546 23504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:59.034934 23504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:59.044610 23509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:59.044608 23512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:59.044623 23510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:59.044958 23504 server_base.cc:1034] running on GCE node
I20260619 22:29:59.045500 23504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:59.046218 23504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:59.047441 23504 hybrid_clock.cc:648] HybridClock initialized: now 1781908199047425 us; error 29 us; skew 500 ppm
I20260619 22:29:59.049432 23504 webserver.cc:460] Webserver started at http://127.5.176.193:44983/ using document root <none> and password file <none>
I20260619 22:29:59.050040 23504 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:59.050103 23504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:59.050372 23504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:59.052511 23504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "1f1989f25993471a8bac3de1e8a6c2c4"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "c81a208b22db663d6273752dfab1beef"
server_key_iv: "90bb25a88e7a766a066bcdbde480c5a2"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.053117 23504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "1f1989f25993471a8bac3de1e8a6c2c4"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "c81a208b22db663d6273752dfab1beef"
server_key_iv: "90bb25a88e7a766a066bcdbde480c5a2"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.057089 23504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:29:59.059758 23518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.061024 23504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:29:59.061178 23504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "1f1989f25993471a8bac3de1e8a6c2c4"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "c81a208b22db663d6273752dfab1beef"
server_key_iv: "90bb25a88e7a766a066bcdbde480c5a2"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.061305 23504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:59.086340 23504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:59.087081 23504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:59.087286 23504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:59.087895 23504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:59.088845 23504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:59.088914 23504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.088987 23504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:59.089031 23504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.098487 23504 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:40911
I20260619 22:29:59.098505 23631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:40911 every 8 connection(s)
I20260619 22:29:59.099440 23504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:29:59.103864 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23504
I20260619 22:29:59.103996 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:29:59.104275 5827 external_mini_cluster.cc:1428] Setting key e2300aa108f14c1748595f07d09b94c5
I20260619 22:29:59.106411 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36785
--builtin_ntp_servers=127.5.176.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:29:59.110813 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.101292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:55789 (local address 127.5.176.254:36785)
0619 22:29:59.101459 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.101463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.102653 (+ 1190us) server_negotiation.cc:407] Connection header received
0619 22:29:59.104318 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.104322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.104387 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.104505 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:59.105589 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.106250 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.108103 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.108334 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.108451 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.110098 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:59.110101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:59.110103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:29:59.110117 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:59.110191 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:29:59.110206 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:59.110283 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.110550 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.110643 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:29:59.111703 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.101665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:36785 (local address 127.5.176.193:55789)
0619 22:29:59.102505 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.102550 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.104091 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.104692 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.104698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.104787 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:59.105388 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.105403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.106387 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.106391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.107958 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.107971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.108501 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.108505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.108715 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.109687 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:29:59.109689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:59.109753 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:29:59.109870 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:59.110296 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:59.110307 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:59.110312 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.110524 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.110842 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":108,"threads_started":1}
I20260619 22:29:59.113310 23632 heartbeater.cc:347] Connected to a master server at 127.5.176.254:36785
I20260619 22:29:59.113581 23632 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:59.114217 23632 heartbeater.cc:511] Master 127.5.176.254:36785 requested a full tablet report, sending...
I20260619 22:29:59.116128 23444 ts_manager.cc:194] Registered new tserver with Master: 1f1989f25993471a8bac3de1e8a6c2c4 (127.5.176.193:40911)
I20260619 22:29:59.117942 23444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.193:55789
W20260619 22:29:59.221993 23635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:59.222277 23635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:59.222318 23635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:59.226394 23635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:59.226459 23635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:59.226516 23635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:29:59.231200 23635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:59.232225 23635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:59.233671 23635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:59.243141 23643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:59.243180 23640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:59.243171 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:59.243224 23635 server_base.cc:1034] running on GCE node
I20260619 22:29:59.243791 23635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:59.244429 23635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:59.245636 23635 hybrid_clock.cc:648] HybridClock initialized: now 1781908199245624 us; error 25 us; skew 500 ppm
I20260619 22:29:59.247489 23635 webserver.cc:460] Webserver started at http://127.5.176.194:43735/ using document root <none> and password file <none>
I20260619 22:29:59.248095 23635 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:59.248188 23635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:59.248422 23635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:59.250582 23635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "f6686c583a8c428ba08fa672269eb0c4"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "8ff769286b12e18b948e027d049b6655"
server_key_iv: "72559814435d9b172a1d6b5c821ea8f2"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.251147 23635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "f6686c583a8c428ba08fa672269eb0c4"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "8ff769286b12e18b948e027d049b6655"
server_key_iv: "72559814435d9b172a1d6b5c821ea8f2"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.254818 23635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:29:59.257290 23649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.258634 23635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:29:59.258771 23635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "f6686c583a8c428ba08fa672269eb0c4"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "8ff769286b12e18b948e027d049b6655"
server_key_iv: "72559814435d9b172a1d6b5c821ea8f2"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.258894 23635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:59.288949 23635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:59.289676 23635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:59.289889 23635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:59.290512 23635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:59.291450 23635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:59.291524 23635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.291602 23635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:59.291653 23635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.300879 23635 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:46253
I20260619 22:29:59.300907 23762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:46253 every 8 connection(s)
I20260619 22:29:59.301782 23635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:29:59.304441 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23635
I20260619 22:29:59.304577 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:29:59.304903 5827 external_mini_cluster.cc:1428] Setting key a5dd43024138cba1bea428572eb14c7f
I20260619 22:29:59.307073 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36785
--builtin_ntp_servers=127.5.176.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:29:59.311920 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.303864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:48883 (local address 127.5.176.254:36785)
0619 22:29:59.304022 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.304026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.305342 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 22:29:59.307059 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.307062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.307116 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.307195 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:59.308081 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.308620 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.309776 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.310005 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.310122 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.311402 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:59.311405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:59.311407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:29:59.311421 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:59.311440 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:29:59.311448 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:59.311528 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.311700 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.311766 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 22:29:59.312904 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.304362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:36785 (local address 127.5.176.194:48883)
0619 22:29:59.305158 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.305201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.306846 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.307363 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.307370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.307453 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:59.307940 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.307952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.308754 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.308762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.309644 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.309653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.310205 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.310211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.310372 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.311112 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:29:59.311114 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:59.311156 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:29:59.311225 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:59.311541 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:59.311555 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:59.311565 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.311689 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.312035 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":140,"threads_started":1}
I20260619 22:29:59.314195 23763 heartbeater.cc:347] Connected to a master server at 127.5.176.254:36785
I20260619 22:29:59.314495 23763 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:59.315215 23763 heartbeater.cc:511] Master 127.5.176.254:36785 requested a full tablet report, sending...
I20260619 22:29:59.316493 23444 ts_manager.cc:194] Registered new tserver with Master: f6686c583a8c428ba08fa672269eb0c4 (127.5.176.194:46253)
I20260619 22:29:59.317090 23444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.194:48883
W20260619 22:29:59.421399 23766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:29:59.421727 23766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:29:59.421805 23766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:29:59.425868 23766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:29:59.425977 23766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:29:59.426072 23766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:29:59.430909 23766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:29:59.432068 23766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:29:59.433454 23766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:29:59.442996 23771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:59.443136 23766 server_base.cc:1034] running on GCE node
W20260619 22:29:59.443002 23772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:29:59.443310 23774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:29:59.443730 23766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:29:59.444422 23766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:29:59.445585 23766 hybrid_clock.cc:648] HybridClock initialized: now 1781908199445571 us; error 29 us; skew 500 ppm
I20260619 22:29:59.447551 23766 webserver.cc:460] Webserver started at http://127.5.176.195:34179/ using document root <none> and password file <none>
I20260619 22:29:59.448148 23766 fs_manager.cc:359] Metadata directory not provided
I20260619 22:29:59.448210 23766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:29:59.448457 23766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:29:59.450533 23766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "1595b0a1c7a5409488ed3fd13d469cfb"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "e3d8c34644622d18a758006b0cc4c752"
server_key_iv: "c8cbe53b69dd040c900d87d3a2aab01d"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.451078 23766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "1595b0a1c7a5409488ed3fd13d469cfb"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "e3d8c34644622d18a758006b0cc4c752"
server_key_iv: "c8cbe53b69dd040c900d87d3a2aab01d"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.454702 23766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:29:59.457100 23780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.458370 23766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:29:59.458503 23766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "1595b0a1c7a5409488ed3fd13d469cfb"
format_stamp: "Formatted at 2026-06-19 22:29:59 on dist-test-slave-bt27"
server_key: "e3d8c34644622d18a758006b0cc4c752"
server_key_iv: "c8cbe53b69dd040c900d87d3a2aab01d"
server_key_version: "encryptionkey@0"
I20260619 22:29:59.458616 23766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:29:59.469411 23766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:29:59.470062 23766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:29:59.470288 23766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:29:59.470854 23766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:29:59.471750 23766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:29:59.471822 23766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.471892 23766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:29:59.471936 23766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:29:59.480564 23766 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:45413
I20260619 22:29:59.480583 23893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:45413 every 8 connection(s)
I20260619 22:29:59.481570 23766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:29:59.483618 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23766
I20260619 22:29:59.483726 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:29:59.484030 5827 external_mini_cluster.cc:1428] Setting key c9f2e96c6e4807328d722a4126eeed78
I20260619 22:29:59.491207 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.483354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:46875 (local address 127.5.176.254:36785)
0619 22:29:59.483491 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.483495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.484797 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 22:29:59.486612 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.486614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.486661 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.486736 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:59.487590 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.488178 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.489362 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.489584 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.489667 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.490794 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:59.490798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:59.490800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:29:59.490812 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:59.490830 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:29:59.490837 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:59.490899 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.491044 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.491095 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:29:59.492079 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.483832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:36785 (local address 127.5.176.195:46875)
0619 22:29:59.484631 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.484671 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.486398 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.486890 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.486897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.486978 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:29:59.487442 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.487454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.488333 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.488336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.489244 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.489254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.489750 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.489753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.489916 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.490542 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:29:59.490544 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:29:59.490589 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:29:59.490653 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:29:59.490924 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:29:59.490930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:29:59.490934 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.491028 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.491243 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":132,"threads_started":1}
I20260619 22:29:59.493152 23894 heartbeater.cc:347] Connected to a master server at 127.5.176.254:36785
I20260619 22:29:59.493346 23894 heartbeater.cc:464] Registering TS with master...
I20260619 22:29:59.493803 23894 heartbeater.cc:511] Master 127.5.176.254:36785 requested a full tablet report, sending...
I20260619 22:29:59.494836 23444 ts_manager.cc:194] Registered new tserver with Master: 1595b0a1c7a5409488ed3fd13d469cfb (127.5.176.195:45413)
I20260619 22:29:59.495373 23444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.195:46875
I20260619 22:29:59.497848 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:29:59.503909 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.500191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.5.176.254:36785)
0619 22:29:59.500316 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.500321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.500478 (+ 157us) server_negotiation.cc:407] Connection header received
0619 22:29:59.500569 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.500571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.500607 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.500674 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:29:59.500955 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.501449 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.502466 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.502685 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.502770 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.503527 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:29:59.503530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:29:59.503533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:29:59.503553 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:29:59.503571 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:29:59.503579 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:29:59.503645 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.503736 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.503782 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:29:59.506904 23444 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:29:59.509970 23444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:29:59.524585 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.518717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:45413 (local address 127.0.0.1:44010)
0619 22:29:59.519230 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.519282 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.519437 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.520061 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.520068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.520094 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.520244 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.520253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.521651 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.521655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.522896 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.522906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.523987 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.523991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.524102 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.524153 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.524258 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.524406 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":63,"threads_started":1}
I20260619 22:29:59.524952 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.519777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.5.176.194:46253)
0619 22:29:59.520189 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.520200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.520220 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:29:59.520291 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.520296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.520467 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.520610 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.520852 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.522017 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.523444 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.524105 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.524208 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.524387 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.524469 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.524586 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":122,"threads_started":1}
I20260619 22:29:59.524950 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.518907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.5.176.195:45413)
0619 22:29:59.519542 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.519554 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.519580 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:29:59.519663 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.519670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.519872 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.520412 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.520616 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.521532 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.523037 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.523856 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.523961 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.524153 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.524668 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.524780 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":156,"threads_started":1}
I20260619 22:29:59.525234 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.519376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:46253 (local address 127.0.0.1:41326)
0619 22:29:59.520091 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.520104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.520178 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.520600 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.520603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.520621 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.520715 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.520724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.522172 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.522176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.523317 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.523326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.524238 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.524242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.524330 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.524355 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.524441 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.524543 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260619 22:29:59.525978 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.518952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40911 (local address 127.0.0.1:38056)
0619 22:29:59.519681 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.519694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.519772 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.520832 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.520835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.520853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.520933 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.520939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.522223 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.522227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.523468 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.523478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.524525 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.524528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.524626 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.524648 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.524974 (+ 326us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.525845 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":36,"thread_start_us":106,"threads_started":1}
I20260619 22:29:59.526645 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.519475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.5.176.193:40911)
0619 22:29:59.519937 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.519951 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.520413 (+ 462us) server_negotiation.cc:407] Connection header received
0619 22:29:59.520498 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.520501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.520674 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.520825 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.521028 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.522117 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.523601 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.524382 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.526080 (+ 1698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.526298 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.526338 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.526465 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":146,"threads_started":1}
I20260619 22:29:59.528083 23828 tablet_service.cc:1467] Processing CreateTablet for tablet 0489417af09d496baf9eb22cc837e55d (DEFAULT_TABLE table=test-table [id=c4fe8e1fdf814492b2416bfd96e29848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:59.528643 23697 tablet_service.cc:1467] Processing CreateTablet for tablet 0489417af09d496baf9eb22cc837e55d (DEFAULT_TABLE table=test-table [id=c4fe8e1fdf814492b2416bfd96e29848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:59.529065 23828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0489417af09d496baf9eb22cc837e55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:59.529094 23566 tablet_service.cc:1467] Processing CreateTablet for tablet 0489417af09d496baf9eb22cc837e55d (DEFAULT_TABLE table=test-table [id=c4fe8e1fdf814492b2416bfd96e29848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:29:59.529639 23697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0489417af09d496baf9eb22cc837e55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:59.530009 23566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0489417af09d496baf9eb22cc837e55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:29:59.535470 23911 tablet_bootstrap.cc:492] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Bootstrap starting.
I20260619 22:29:59.536417 23912 tablet_bootstrap.cc:492] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Bootstrap starting.
I20260619 22:29:59.537139 23913 tablet_bootstrap.cc:492] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: Bootstrap starting.
I20260619 22:29:59.537905 23911 tablet_bootstrap.cc:654] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:59.538457 23912 tablet_bootstrap.cc:654] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:59.538790 23911 log.cc:824] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:59.538851 23913 tablet_bootstrap.cc:654] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: Neither blocks nor log segments found. Creating new log.
I20260619 22:29:59.539218 23912 log.cc:824] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:59.539626 23913 log.cc:824] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:29:59.541131 23911 tablet_bootstrap.cc:492] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: No bootstrap required, opened a new log
I20260619 22:29:59.541149 23912 tablet_bootstrap.cc:492] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: No bootstrap required, opened a new log
I20260619 22:29:59.541332 23911 ts_tablet_manager.cc:1397] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:29:59.541368 23912 ts_tablet_manager.cc:1397] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:29:59.541600 23913 tablet_bootstrap.cc:492] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: No bootstrap required, opened a new log
I20260619 22:29:59.541771 23913 ts_tablet_manager.cc:1397] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:29:59.544899 23911 raft_consensus.cc:348] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.545142 23911 raft_consensus.cc:374] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:59.545193 23911 raft_consensus.cc:729] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1595b0a1c7a5409488ed3fd13d469cfb, State: Initialized, Role: FOLLOWER
I20260619 22:29:59.545637 23911 consensus_queue.cc:260] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.545696 23913 raft_consensus.cc:348] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.545696 23912 raft_consensus.cc:348] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.545934 23913 raft_consensus.cc:374] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:59.545934 23912 raft_consensus.cc:374] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:29:59.546001 23912 raft_consensus.cc:729] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6686c583a8c428ba08fa672269eb0c4, State: Initialized, Role: FOLLOWER
I20260619 22:29:59.546010 23913 raft_consensus.cc:729] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1989f25993471a8bac3de1e8a6c2c4, State: Initialized, Role: FOLLOWER
I20260619 22:29:59.546427 23912 consensus_queue.cc:260] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.546466 23913 consensus_queue.cc:260] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.546514 23894 heartbeater.cc:503] Master 127.5.176.254:36785 was elected leader, sending a full tablet report...
I20260619 22:29:59.547345 23913 ts_tablet_manager.cc:1428] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:29:59.547344 23912 ts_tablet_manager.cc:1428] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:29:59.547590 23632 heartbeater.cc:503] Master 127.5.176.254:36785 was elected leader, sending a full tablet report...
I20260619 22:29:59.548384 23763 heartbeater.cc:503] Master 127.5.176.254:36785 was elected leader, sending a full tablet report...
I20260619 22:29:59.549953 23911 ts_tablet_manager.cc:1428] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260619 22:29:59.552691 23764 tablet.cc:2369] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:29:59.600618 23633 tablet.cc:2369] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:29:59.607766 23919 raft_consensus.cc:482] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:29:59.608024 23919 raft_consensus.cc:504] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.609205 23919 leader_election.cc:288] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1595b0a1c7a5409488ed3fd13d469cfb (127.5.176.195:45413), 1f1989f25993471a8bac3de1e8a6c2c4 (127.5.176.193:40911)
I20260619 22:29:59.612048 23917 raft_consensus.cc:482] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:29:59.612257 23917 raft_consensus.cc:504] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.613379 23917 leader_election.cc:288] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f1989f25993471a8bac3de1e8a6c2c4 (127.5.176.193:40911), f6686c583a8c428ba08fa672269eb0c4 (127.5.176.194:46253)
I20260619 22:29:59.613668 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.609607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:35645 (local address 127.5.176.195:45413)
0619 22:29:59.609729 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.609734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.610102 (+ 368us) server_negotiation.cc:407] Connection header received
0619 22:29:59.610227 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.610230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.610278 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.610361 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.610683 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.611239 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.612569 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.613138 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.613242 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.613283 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.613483 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.613549 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:29:59.613695 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.609524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:45413 (local address 127.5.176.194:35645)
0619 22:29:59.609980 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.609995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.610112 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.610388 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.610390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.610404 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.610548 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.610555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.611347 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.611349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.612400 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.612410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.613242 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.613245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.613353 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.613372 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.613456 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.613526 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":31,"thread_start_us":128,"threads_started":1}
I20260619 22:29:59.613792 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.609607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:35593 (local address 127.5.176.193:40911)
0619 22:29:59.609729 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.609734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.609833 (+ 99us) server_negotiation.cc:407] Connection header received
0619 22:29:59.609993 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.609998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.610065 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.610192 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.610482 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.611072 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.612342 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.612972 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.613098 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.613149 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.613598 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.613681 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:29:59.613545 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.609526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40911 (local address 127.5.176.194:35593)
0619 22:29:59.609738 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.609753 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.609873 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.610197 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.610200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.610217 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.610355 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.610374 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.611178 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.611181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.612188 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.612199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.613109 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.613112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.613208 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.613233 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.613311 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.613400 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 22:29:59.614365 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0489417af09d496baf9eb22cc837e55d" candidate_uuid: "f6686c583a8c428ba08fa672269eb0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" is_pre_election: true
I20260619 22:29:59.614578 23848 raft_consensus.cc:2454] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6686c583a8c428ba08fa672269eb0c4 in term 0.
I20260619 22:29:59.614595 23586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0489417af09d496baf9eb22cc837e55d" candidate_uuid: "f6686c583a8c428ba08fa672269eb0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" is_pre_election: true
I20260619 22:29:59.614861 23586 raft_consensus.cc:2454] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6686c583a8c428ba08fa672269eb0c4 in term 0.
I20260619 22:29:59.615406 23652 leader_election.cc:302] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1595b0a1c7a5409488ed3fd13d469cfb, 1f1989f25993471a8bac3de1e8a6c2c4, f6686c583a8c428ba08fa672269eb0c4; no voters:
I20260619 22:29:59.615653 23919 raft_consensus.cc:2781] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:29:59.615800 23919 raft_consensus.cc:482] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:29:59.615906 23919 raft_consensus.cc:3037] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:59.617260 23919 raft_consensus.cc:504] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.617651 23919 leader_election.cc:288] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [CANDIDATE]: Term 1 election: Requested vote from peers 1595b0a1c7a5409488ed3fd13d469cfb (127.5.176.195:45413), 1f1989f25993471a8bac3de1e8a6c2c4 (127.5.176.193:40911)
I20260619 22:29:59.617949 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0489417af09d496baf9eb22cc837e55d" candidate_uuid: "f6686c583a8c428ba08fa672269eb0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1595b0a1c7a5409488ed3fd13d469cfb"
I20260619 22:29:59.618062 23848 raft_consensus.cc:3037] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:59.618091 23586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0489417af09d496baf9eb22cc837e55d" candidate_uuid: "f6686c583a8c428ba08fa672269eb0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1989f25993471a8bac3de1e8a6c2c4"
I20260619 22:29:59.618227 23586 raft_consensus.cc:3037] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:29:59.618829 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.613686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40911 (local address 127.5.176.195:44355)
0619 22:29:59.613867 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.613882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.613960 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.614793 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.614799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.614819 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.614917 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.614924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.616170 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.616174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.617346 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.617356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.618397 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.618401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.618502 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.618536 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.618605 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.618674 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 22:29:59.619002 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.614477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:37051 (local address 127.5.176.194:46253)
0619 22:29:59.614889 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.614895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.614907 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:29:59.614946 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.614952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.615006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.615067 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.615475 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.616167 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.617807 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.618385 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.618463 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.618504 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.618806 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.618899 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260619 22:29:59.619153 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.614368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:44355 (local address 127.5.176.193:40911)
0619 22:29:59.614584 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.614588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.614601 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:29:59.614641 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.614644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.614680 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.614759 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:29:59.615306 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.616051 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.617571 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.618163 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.618267 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.618314 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.618963 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.619042 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 22:29:59.619309 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.613920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:46253 (local address 127.5.176.195:37051)
0619 22:29:59.614379 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.614394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:29:59.614484 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:29:59.615160 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:29:59.615163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:29:59.615179 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:29:59.615288 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.615355 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.616273 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.616276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.617684 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:29:59.617693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.618589 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.618596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:29:59.618682 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.618700 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:29:59.619105 (+ 405us) client_negotiation.cc:240] Negotiation successful
0619 22:29:59.619172 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":126,"threads_started":1}
I20260619 22:29:59.619362 23848 raft_consensus.cc:2454] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6686c583a8c428ba08fa672269eb0c4 in term 1.
I20260619 22:29:59.619427 23586 raft_consensus.cc:2454] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6686c583a8c428ba08fa672269eb0c4 in term 1.
I20260619 22:29:59.619724 23651 leader_election.cc:302] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1595b0a1c7a5409488ed3fd13d469cfb, f6686c583a8c428ba08fa672269eb0c4; no voters:
I20260619 22:29:59.619957 23919 raft_consensus.cc:2781] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:29:59.619973 23586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0489417af09d496baf9eb22cc837e55d" candidate_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" is_pre_election: true
I20260619 22:29:59.620232 23586 raft_consensus.cc:2379] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1595b0a1c7a5409488ed3fd13d469cfb in current term 1: Already voted for candidate f6686c583a8c428ba08fa672269eb0c4 in this term.
I20260619 22:29:59.620177 23919 raft_consensus.cc:686] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 1 LEADER]: Becoming Leader. State: Replica: f6686c583a8c428ba08fa672269eb0c4, State: Running, Role: LEADER
I20260619 22:29:59.620165 23717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0489417af09d496baf9eb22cc837e55d" candidate_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6686c583a8c428ba08fa672269eb0c4" is_pre_election: true
I20260619 22:29:59.620549 23919 consensus_queue.cc:237] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } }
I20260619 22:29:59.620975 23783 leader_election.cc:302] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1595b0a1c7a5409488ed3fd13d469cfb; no voters: 1f1989f25993471a8bac3de1e8a6c2c4, f6686c583a8c428ba08fa672269eb0c4
I20260619 22:29:59.621268 23917 raft_consensus.cc:2726] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 22:29:59.623745 23443 catalog_manager.cc:5495] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6686c583a8c428ba08fa672269eb0c4 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "f6686c583a8c428ba08fa672269eb0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6686c583a8c428ba08fa672269eb0c4" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 46253 } health_report { overall_health: HEALTHY } } }
I20260619 22:29:59.650832 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.646840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.5.176.194:46253)
0619 22:29:59.646990 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:29:59.646995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:29:59.647011 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:29:59.647116 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:29:59.647118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:29:59.647173 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:29:59.647279 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:29:59.647557 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.648435 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.649290 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:29:59.649530 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:29:59.649611 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:29:59.649934 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:29:59.650468 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:29:59.650573 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 22:29:59.650646 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 22:29:59.650697 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:29:59.656587 23586 raft_consensus.cc:1261] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 1 FOLLOWER]: Refusing update from remote peer f6686c583a8c428ba08fa672269eb0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:29:59.656625 23848 raft_consensus.cc:1261] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 1 FOLLOWER]: Refusing update from remote peer f6686c583a8c428ba08fa672269eb0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:29:59.657294 23919 consensus_queue.cc:1035] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1989f25993471a8bac3de1e8a6c2c4" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:29:59.657512 23922 consensus_queue.cc:1035] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1595b0a1c7a5409488ed3fd13d469cfb" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:29:59.668103 23926 mvcc.cc:205] Tried to move back new op lower bound from 7298695985786445824 to 7298695985648680960. Current Snapshot: MvccSnapshot[applied={T|T < 7298695985786445824 or (T in {7298695985786445824})}]
I20260619 22:29:59.675406 23444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:29:59.675618 23444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:29:59.678318 23444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6997a8b3391f4e408e5d0260dec73b28: Sending DeleteTablet for 3 replicas of tablet 0489417af09d496baf9eb22cc837e55d
I20260619 22:29:59.679116 23828 tablet_service.cc:1514] Processing DeleteTablet for tablet 0489417af09d496baf9eb22cc837e55d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:59 UTC) from {username='slave'} at 127.0.0.1:44010
I20260619 22:29:59.679188 23566 tablet_service.cc:1514] Processing DeleteTablet for tablet 0489417af09d496baf9eb22cc837e55d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:59 UTC) from {username='slave'} at 127.0.0.1:38056
I20260619 22:29:59.679250 23697 tablet_service.cc:1514] Processing DeleteTablet for tablet 0489417af09d496baf9eb22cc837e55d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:29:59 UTC) from {username='slave'} at 127.0.0.1:41326
I20260619 22:29:59.679806 23936 tablet_replica.cc:330] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: stopping tablet replica
I20260619 22:29:59.679808 23935 tablet_replica.cc:330] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: stopping tablet replica
I20260619 22:29:59.680112 23934 tablet_replica.cc:330] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4: stopping tablet replica
I20260619 22:29:59.680177 23935 raft_consensus.cc:2229] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:59.680408 23934 raft_consensus.cc:2229] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:29:59.680509 23935 raft_consensus.cc:2258] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:59.680554 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23504
I20260619 22:29:59.680714 23934 raft_consensus.cc:2258] T 0489417af09d496baf9eb22cc837e55d P 1f1989f25993471a8bac3de1e8a6c2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:59.682312 23935 ts_tablet_manager.cc:1905] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:29:59.682895 23936 raft_consensus.cc:2229] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:29:59.683314 23936 raft_consensus.cc:2258] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:29:59.684952 23936 ts_tablet_manager.cc:1905] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:29:59.685276 23935 ts_tablet_manager.cc:1918] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:29:59.685388 23935 log.cc:1196] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/0489417af09d496baf9eb22cc837e55d
I20260619 22:29:59.685730 23935 ts_tablet_manager.cc:1939] T 0489417af09d496baf9eb22cc837e55d P 1595b0a1c7a5409488ed3fd13d469cfb: Deleting consensus metadata
I20260619 22:29:59.687108 23429 catalog_manager.cc:4841] TS 1595b0a1c7a5409488ed3fd13d469cfb (127.5.176.195:45413): tablet 0489417af09d496baf9eb22cc837e55d (table test-table [id=c4fe8e1fdf814492b2416bfd96e29848]) successfully deleted
I20260619 22:29:59.687925 23936 ts_tablet_manager.cc:1918] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:29:59.688023 23936 log.cc:1196] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/0489417af09d496baf9eb22cc837e55d
I20260619 22:29:59.688340 23936 ts_tablet_manager.cc:1939] T 0489417af09d496baf9eb22cc837e55d P f6686c583a8c428ba08fa672269eb0c4: Deleting consensus metadata
W20260619 22:29:59.688426 23430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:40911 (error 108)
I20260619 22:29:59.688681 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23635
I20260619 22:29:59.690692 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.690323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40911 (local address 127.0.0.1:38070)
0619 22:29:59.690494 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.690581 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 22:29:59.690886 23430 catalog_manager.cc:4584] TS 1f1989f25993471a8bac3de1e8a6c2c4 (127.5.176.193:40911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0489417af09d496baf9eb22cc837e55d: Network error: Client connection negotiation failed: client connection to 127.5.176.193:40911: connect: Connection refused (error 111)
I20260619 22:29:59.695294 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23766
I20260619 22:29:59.696704 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:29:59.696290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:46253 (local address 127.0.0.1:41342)
0619 22:29:59.696514 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 22:29:59.696582 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.194:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 22:29:59.701215 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23413
2026-06-19T22:29:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:29:59 dist-test-slave-bt27 krb5kdc[23940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:29:59 dist-test-slave-bt27 krb5kdc[23940](info): set up 2 sockets
Jun 19 22:29:59 dist-test-slave-bt27 krb5kdc[23940](info): commencing operation
krb5kdc: starting...
W20260619 22:30:00.742057 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:00 dist-test-slave-bt27 krb5kdc[23940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:00.868993 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:33797
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:33797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:00.983024 23953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:00.983382 23953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:00.983462 23953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:00.987960 23953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:00.988059 23953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:00.988097 23953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:00.988137 23953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:00.988173 23953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:00.993845 23953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:33797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:00.995083 23953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:00.996120 23953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:01.005404 23958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:01.005405 23959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:01.005640 23953 server_base.cc:1034] running on GCE node
W20260619 22:30:01.005786 23961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:01.006592 23953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:01.008239 23953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:01.009542 23953 hybrid_clock.cc:648] HybridClock initialized: now 1781908201009511 us; error 53 us; skew 500 ppm
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:01.013172 23953 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:01.014333 23953 webserver.cc:460] Webserver started at http://127.5.176.254:33575/ using document root <none> and password file <none>
I20260619 22:30:01.014946 23953 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:01.015043 23953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:01.015288 23953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:01.017503 23953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "35d275c98c614755b6c0a54984c733db"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "7913ac50703aeb88e28f03f528afa6eb"
server_key_iv: "46ab03f914910add791347ede349f205"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.018355 23953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "35d275c98c614755b6c0a54984c733db"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "7913ac50703aeb88e28f03f528afa6eb"
server_key_iv: "46ab03f914910add791347ede349f205"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.022670 23953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:30:01.025705 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.027344 23953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:01.027539 23953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "35d275c98c614755b6c0a54984c733db"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "7913ac50703aeb88e28f03f528afa6eb"
server_key_iv: "46ab03f914910add791347ede349f205"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.027685 23953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:01.043942 23953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:01.047184 23953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:01.047416 23953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:01.054769 23953 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:33797
I20260619 22:30:01.054801 24020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:33797 every 8 connection(s)
I20260619 22:30:01.055851 23953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:01.058779 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:01.064796 24021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db: Bootstrap starting.
I20260619 22:30:01.065834 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 23953
I20260619 22:30:01.065974 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:01.066264 5827 external_mini_cluster.cc:1428] Setting key 5339867a5a10c1a2c8a529df02858cc1
I20260619 22:30:01.067760 24021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:01.068486 24021 log.cc:824] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:01.070569 24021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db: No bootstrap required, opened a new log
I20260619 22:30:01.074291 24021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d275c98c614755b6c0a54984c733db" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33797 } }
I20260619 22:30:01.074556 24021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:01.074651 24021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d275c98c614755b6c0a54984c733db, State: Initialized, Role: FOLLOWER
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:01.075168 24021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d275c98c614755b6c0a54984c733db" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33797 } }
I20260619 22:30:01.075338 24021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:01.075409 24021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:01.075519 24021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:01.076721 24021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d275c98c614755b6c0a54984c733db" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33797 } }
I20260619 22:30:01.077104 24021 leader_election.cc:302] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d275c98c614755b6c0a54984c733db; no voters:
I20260619 22:30:01.077497 24021 leader_election.cc:288] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:01.077580 24026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:01.077756 24026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [term 1 LEADER]: Becoming Leader. State: Replica: 35d275c98c614755b6c0a54984c733db, State: Running, Role: LEADER
I20260619 22:30:01.078120 24026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d275c98c614755b6c0a54984c733db" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33797 } }
I20260619 22:30:01.078847 24021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:01.080008 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35d275c98c614755b6c0a54984c733db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d275c98c614755b6c0a54984c733db" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33797 } } }
I20260619 22:30:01.080022 24027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d275c98c614755b6c0a54984c733db. Latest consensus state: current_term: 1 leader_uuid: "35d275c98c614755b6c0a54984c733db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d275c98c614755b6c0a54984c733db" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33797 } } }
I20260619 22:30:01.080220 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:01.080236 24027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:01.080632 24035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:01.081005 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.067410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.5.176.254:33797)
0619 22:30:01.067874 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.067895 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.067943 (+ 48us) server_negotiation.cc:407] Connection header received
0619 22:30:01.068646 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.068669 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.069080 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.069484 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:01.070338 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.071388 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.072258 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.072748 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.072928 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.075716 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:01.075721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:01.075734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:01.075763 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:01.077665 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.078203 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.078208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.078214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.078346 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.078615 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.078619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.078621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.079080 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:01.079297 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.079572 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.079955 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260619 22:30:01.084381 24035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:01.094592 24035 catalog_manager.cc:1347] Generated new cluster ID: 45ccdb8a332842a68d4fab7763807d2b
I20260619 22:30:01.094702 24035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:01.107385 24035 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:01.108502 24035 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:01.115510 24035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db: Generated new TSK 0
I20260619 22:30:01.116670 24035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:30:01.146875 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33797
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:01.262094 24047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:01.262498 24047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:01.262581 24047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:01.266865 24047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:01.266932 24047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:01.267019 24047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:30:01.272621 24047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:01.273775 24047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:01.274873 24047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:01.285404 24055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:01.285415 24052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:01.285625 24053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:01.285729 24047 server_base.cc:1034] running on GCE node
I20260619 22:30:01.286592 24047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:01.287442 24047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:01.288689 24047 hybrid_clock.cc:648] HybridClock initialized: now 1781908201288650 us; error 52 us; skew 500 ppm
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:01.296525 24047 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:30:01.297662 24047 webserver.cc:460] Webserver started at http://127.5.176.193:45691/ using document root <none> and password file <none>
I20260619 22:30:01.298408 24047 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:01.298519 24047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:01.298908 24047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:01.301415 24047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "b25680dd0965488191d8536392255f0c"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "3d1b0dcf0062a603445890c8d57b4980"
server_key_iv: "32c92f295f6e4dc7f207960a66ebdebe"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.302006 24047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "b25680dd0965488191d8536392255f0c"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "3d1b0dcf0062a603445890c8d57b4980"
server_key_iv: "32c92f295f6e4dc7f207960a66ebdebe"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.306514 24047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:01.309161 24062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.310664 24047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:01.310773 24047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "b25680dd0965488191d8536392255f0c"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "3d1b0dcf0062a603445890c8d57b4980"
server_key_iv: "32c92f295f6e4dc7f207960a66ebdebe"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.310885 24047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:01.321672 24047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:01.325948 24047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:01.326122 24047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:01.326792 24047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:01.327716 24047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:01.327765 24047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.327806 24047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:01.327822 24047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.338687 24047 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:34597
I20260619 22:30:01.338713 24175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:34597 every 8 connection(s)
I20260619 22:30:01.339689 24047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:30:01.344133 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24047
I20260619 22:30:01.344264 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:30:01.344674 5827 external_mini_cluster.cc:1428] Setting key 173127e52a488c296e72bae2ff5163aa
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
I20260619 22:30:01.355075 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.341803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:39119 (local address 127.5.176.254:33797)
0619 22:30:01.341989 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.341993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.343342 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 22:30:01.344386 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.344392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.344458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.344587 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:01.346021 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.346693 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.347901 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.348142 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.348255 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.351044 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:01.351048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:01.351050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:01.351076 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:01.352619 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.353282 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.353285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.353286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.353365 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.353790 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.353792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.353793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.354008 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:01.354087 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.354833 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.354950 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 22:30:01.355885 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.342266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33797 (local address 127.5.176.193:39119)
0619 22:30:01.343147 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:01.343192 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:01.344099 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:01.344814 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:01.344823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:01.345311 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:01.345855 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.345871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.346837 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.346841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.347765 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.347775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.348301 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.348306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.348484 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.349147 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:01.349163 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:01.350827 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:01.352748 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:01.352754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:01.352773 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:01.353172 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:01.353469 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:01.353472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:01.353476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:01.353677 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:01.354123 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:01.354132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:01.354533 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 22:30:01.354736 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 22:30:01.355014 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:30:01.357347 24176 heartbeater.cc:347] Connected to a master server at 127.5.176.254:33797
I20260619 22:30:01.357620 24176 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:01.358309 24176 heartbeater.cc:511] Master 127.5.176.254:33797 requested a full tablet report, sending...
I20260619 22:30:01.360576 23985 ts_manager.cc:194] Registered new tserver with Master: b25680dd0965488191d8536392255f0c (127.5.176.193:34597)
I20260619 22:30:01.362387 23985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:39119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:30:01.375245 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33797
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:01.488945 24181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:01.489208 24181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:01.489252 24181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:01.493417 24181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:01.493479 24181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:01.493538 24181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:30:01.498838 24181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:01.499842 24181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:01.500799 24181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:01.510429 24187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:01.510502 24181 server_base.cc:1034] running on GCE node
W20260619 22:30:01.510429 24186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:01.510455 24189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:01.511201 24181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:01.511934 24181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:01.513127 24181 hybrid_clock.cc:648] HybridClock initialized: now 1781908201513096 us; error 46 us; skew 500 ppm
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:01.516175 24181 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:30:01.517117 24181 webserver.cc:460] Webserver started at http://127.5.176.194:41485/ using document root <none> and password file <none>
I20260619 22:30:01.517722 24181 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:01.517817 24181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:01.518059 24181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:01.520243 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "132d8d2f4640463d9b2acdccc94ded44"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "0ced98da24b6fd885cb6560a502b6cbf"
server_key_iv: "dff6d05835a1418ef8883a4f87d0dc21"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.520867 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "132d8d2f4640463d9b2acdccc94ded44"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "0ced98da24b6fd885cb6560a502b6cbf"
server_key_iv: "dff6d05835a1418ef8883a4f87d0dc21"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.525117 24181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 22:30:01.527750 24196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.529322 24181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 22:30:01.529455 24181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "132d8d2f4640463d9b2acdccc94ded44"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "0ced98da24b6fd885cb6560a502b6cbf"
server_key_iv: "dff6d05835a1418ef8883a4f87d0dc21"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.529582 24181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:01.541404 24181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:01.544890 24181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:01.545111 24181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:01.545730 24181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:01.546693 24181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:01.546747 24181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.546788 24181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:01.546804 24181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.555562 24181 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:34121
I20260619 22:30:01.555625 24309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:34121 every 8 connection(s)
I20260619 22:30:01.556490 24181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:30:01.562273 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24181
I20260619 22:30:01.562425 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:30:01.562732 5827 external_mini_cluster.cc:1428] Setting key 26c7b2f00e9cd7a2769c7c207a014695
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:01.570168 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.558404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52385 (local address 127.5.176.254:33797)
0619 22:30:01.558545 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.558549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.559347 (+ 798us) server_negotiation.cc:407] Connection header received
0619 22:30:01.560234 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.560237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.560293 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.560411 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:01.561535 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.562136 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.563552 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.563762 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.563845 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.566619 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:01.566622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:01.566625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:01.566648 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:01.568118 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.568629 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.568632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.568634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.568715 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.569063 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.569067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.569069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.569254 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:01.569390 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.569853 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.569974 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:30:01.570968 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.558763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33797 (local address 127.5.176.194:52385)
0619 22:30:01.559201 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:01.559241 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:01.560033 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:01.560571 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:01.560578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:01.560933 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:01.561388 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.561401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.562268 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.562271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.563416 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.563427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.563919 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.563926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.564168 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.564845 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:01.564861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:01.566446 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:01.568231 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:01.568237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:01.568252 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:01.568513 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:01.568811 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:01.568814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:01.568816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:01.568963 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:01.569360 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:01.569366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:01.569617 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 22:30:01.569802 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 22:30:01.570163 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:30:01.572281 24310 heartbeater.cc:347] Connected to a master server at 127.5.176.254:33797
I20260619 22:30:01.572520 24310 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:01.573050 24310 heartbeater.cc:511] Master 127.5.176.254:33797 requested a full tablet report, sending...
I20260619 22:30:01.574072 23985 ts_manager.cc:194] Registered new tserver with Master: 132d8d2f4640463d9b2acdccc94ded44 (127.5.176.194:34121)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:30:01.574810 23985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:52385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:30:01.590384 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33797
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:01.707325 24315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:01.707583 24315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:01.707628 24315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:01.711740 24315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:01.711792 24315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:01.711853 24315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:30:01.717435 24315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:01.718578 24315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:01.719578 24315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:01.729733 24321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:01.729686 24320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:01.729684 24323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:01.729966 24315 server_base.cc:1034] running on GCE node
I20260619 22:30:01.730512 24315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:01.731215 24315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:01.732424 24315 hybrid_clock.cc:648] HybridClock initialized: now 1781908201732407 us; error 30 us; skew 500 ppm
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:01.735467 24315 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:30:01.736433 24315 webserver.cc:460] Webserver started at http://127.5.176.195:33885/ using document root <none> and password file <none>
I20260619 22:30:01.737030 24315 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:01.737121 24315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:01.737361 24315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:01.739573 24315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "7b833b0efbaa403f93803735123f5fed"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "80dcefa0cdf1c810d369aca5e1def5be"
server_key_iv: "61036e19b7ad4fa9bbbbc9db001cf709"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.740175 24315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "7b833b0efbaa403f93803735123f5fed"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "80dcefa0cdf1c810d369aca5e1def5be"
server_key_iv: "61036e19b7ad4fa9bbbbc9db001cf709"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.744170 24315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:01.746753 24330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.748027 24315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:01.748170 24315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "7b833b0efbaa403f93803735123f5fed"
format_stamp: "Formatted at 2026-06-19 22:30:01 on dist-test-slave-bt27"
server_key: "80dcefa0cdf1c810d369aca5e1def5be"
server_key_iv: "61036e19b7ad4fa9bbbbc9db001cf709"
server_key_version: "encryptionkey@0"
I20260619 22:30:01.748293 24315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:01.759157 24315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:01.762661 24315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:01.762879 24315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:01.763526 24315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:01.764403 24315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:01.764484 24315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.764573 24315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:01.764626 24315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:01.774318 24315 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:39605
I20260619 22:30:01.774333 24443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:39605 every 8 connection(s)
I20260619 22:30:01.775264 24315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:30:01.777130 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24315
I20260619 22:30:01.777303 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:30:01.777602 5827 external_mini_cluster.cc:1428] Setting key aaf6c58ae7dbe23af943868fcbf4df94
Jun 19 22:30:01 dist-test-slave-bt27 krb5kdc[23940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:01.789263 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.777221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:42831 (local address 127.5.176.254:33797)
0619 22:30:01.777370 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.777374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.778488 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 22:30:01.779727 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.779730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.779784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.779876 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:01.781164 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.781711 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.782952 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.783169 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.783281 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.785779 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:01.785782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:01.785784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:01.785808 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:01.787365 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.787851 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.787855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.787857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.787980 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.788296 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.788298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.788299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.788473 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:01.788546 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.789005 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.789139 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 22:30:01.790071 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.777637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33797 (local address 127.5.176.195:42831)
0619 22:30:01.778262 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:01.778322 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:01.779516 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:01.780063 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:01.780069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:01.780528 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:01.781020 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.781034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.781841 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.781845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.782814 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.782824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.783320 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.783324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.783505 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.784065 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:01.784081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:01.785611 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:01.787472 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:01.787477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:01.787491 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:01.787742 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:01.788068 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:01.788070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:01.788072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:01.788214 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:01.788558 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:01.788563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:01.788795 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 22:30:01.788953 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 22:30:01.789219 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":143,"threads_started":1}
I20260619 22:30:01.791278 24444 heartbeater.cc:347] Connected to a master server at 127.5.176.254:33797
I20260619 22:30:01.791539 24444 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:01.792022 24444 heartbeater.cc:511] Master 127.5.176.254:33797 requested a full tablet report, sending...
I20260619 22:30:01.793017 23984 ts_manager.cc:194] Registered new tserver with Master: 7b833b0efbaa403f93803735123f5fed (127.5.176.195:39605)
I20260619 22:30:01.793557 23984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:42831
I20260619 22:30:01.803591 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:30:01.813910 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.805541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.5.176.254:33797)
0619 22:30:01.805679 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.805684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.805861 (+ 177us) server_negotiation.cc:407] Connection header received
0619 22:30:01.806043 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.806052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.806095 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.806200 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:01.806921 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.807447 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.808298 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.808556 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.808670 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.810325 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:01.810329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:01.810336 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:01.810376 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:01.811783 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.812545 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.812549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.812551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.812633 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:01.812963 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:01.812967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:01.812968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:01.813179 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:01.813335 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.813645 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.813741 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:30:01.817219 23984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:30:01.820339 23984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:30:01.835320 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.829135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:34121 (local address 127.0.0.1:59162)
0619 22:30:01.829502 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:01.829548 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:01.829855 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:01.830738 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:01.830747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:01.830776 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:01.830922 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.830930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.832238 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.832243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.833592 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.833602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.834743 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.834746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.834863 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.834907 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 22:30:01.835029 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 22:30:01.835156 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
I20260619 22:30:01.835376 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.829272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.5.176.194:34121)
0619 22:30:01.829650 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.829666 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.829697 (+ 31us) server_negotiation.cc:407] Connection header received
0619 22:30:01.830086 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.830095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.830349 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.830525 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:01.831075 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.832108 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.833732 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.834631 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.834750 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.834907 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.835102 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.835216 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":149,"threads_started":1}
I20260619 22:30:01.836252 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.830647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34597 (local address 127.0.0.1:36688)
0619 22:30:01.830947 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:01.830963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:01.831058 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:01.831597 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:01.831600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:01.831617 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:01.831711 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.831717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.833412 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.833418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.834577 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.834584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.835796 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.835800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.835914 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.835932 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:30:01.836030 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 22:30:01.836109 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260619 22:30:01.836283 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.830394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.5.176.193:34597)
0619 22:30:01.831024 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.831039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.831065 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:30:01.831158 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.831162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.831350 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.831499 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:01.831950 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.833259 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.834692 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.835524 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.835631 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.835794 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.836061 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.836151 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":116,"threads_started":1}
I20260619 22:30:01.837103 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.829867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:39605 (local address 127.0.0.1:45310)
0619 22:30:01.830879 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:01.830895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:01.830996 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:01.831505 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:01.831508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:01.831527 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:01.831611 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.831617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.833038 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.833041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.834367 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:01.834378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.836661 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.836666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:01.836790 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.836811 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 22:30:01.836875 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 22:30:01.836951 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":110,"threads_started":1}
I20260619 22:30:01.837504 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:01.830289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.5.176.195:39605)
0619 22:30:01.830858 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:01.830869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:01.831093 (+ 224us) server_negotiation.cc:407] Connection header received
0619 22:30:01.831163 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:01.831167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:01.831350 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:01.831499 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:01.831708 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.832817 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.834505 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:01.835362 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:01.836439 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:01.836596 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:01.837284 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 22:30:01.837393 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":132,"threads_started":1}
I20260619 22:30:01.838735 24244 tablet_service.cc:1467] Processing CreateTablet for tablet 1f60fb23f14c47e38dda9e629fa65f3a (DEFAULT_TABLE table=test-table [id=669c3dc48a804a82bf68231944615233]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:01.839958 24244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f60fb23f14c47e38dda9e629fa65f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:01.841113 24110 tablet_service.cc:1467] Processing CreateTablet for tablet 1f60fb23f14c47e38dda9e629fa65f3a (DEFAULT_TABLE table=test-table [id=669c3dc48a804a82bf68231944615233]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:01.841631 24378 tablet_service.cc:1467] Processing CreateTablet for tablet 1f60fb23f14c47e38dda9e629fa65f3a (DEFAULT_TABLE table=test-table [id=669c3dc48a804a82bf68231944615233]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:01.842242 24110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f60fb23f14c47e38dda9e629fa65f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:01.842633 24378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f60fb23f14c47e38dda9e629fa65f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:01.846875 24461 tablet_bootstrap.cc:492] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Bootstrap starting.
I20260619 22:30:01.849017 24461 tablet_bootstrap.cc:654] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:01.849646 24462 tablet_bootstrap.cc:492] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Bootstrap starting.
I20260619 22:30:01.850085 24463 tablet_bootstrap.cc:492] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: Bootstrap starting.
I20260619 22:30:01.850630 24461 log.cc:824] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:01.852474 24463 tablet_bootstrap.cc:654] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:01.852795 24461 tablet_bootstrap.cc:492] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: No bootstrap required, opened a new log
I20260619 22:30:01.853135 24461 ts_tablet_manager.cc:1397] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:30:01.853914 24463 log.cc:824] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:01.854081 24462 tablet_bootstrap.cc:654] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:01.855269 24462 log.cc:824] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:01.856179 24463 tablet_bootstrap.cc:492] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: No bootstrap required, opened a new log
I20260619 22:30:01.856405 24463 ts_tablet_manager.cc:1397] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:30:01.856900 24461 raft_consensus.cc:348] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:01.857336 24461 raft_consensus.cc:374] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:01.857405 24461 raft_consensus.cc:729] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132d8d2f4640463d9b2acdccc94ded44, State: Initialized, Role: FOLLOWER
I20260619 22:30:01.857538 24462 tablet_bootstrap.cc:492] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: No bootstrap required, opened a new log
I20260619 22:30:01.857802 24462 ts_tablet_manager.cc:1397] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 22:30:01.857949 24461 consensus_queue.cc:260] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:01.859062 24461 ts_tablet_manager.cc:1428] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:30:01.859452 24310 heartbeater.cc:503] Master 127.5.176.254:33797 was elected leader, sending a full tablet report...
I20260619 22:30:01.860414 24463 raft_consensus.cc:348] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:01.860702 24463 raft_consensus.cc:374] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:01.860759 24463 raft_consensus.cc:729] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25680dd0965488191d8536392255f0c, State: Initialized, Role: FOLLOWER
I20260619 22:30:01.861227 24463 consensus_queue.cc:260] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:01.861979 24176 heartbeater.cc:503] Master 127.5.176.254:33797 was elected leader, sending a full tablet report...
I20260619 22:30:01.862386 24463 ts_tablet_manager.cc:1428] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 22:30:01.862396 24462 raft_consensus.cc:348] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:01.862666 24462 raft_consensus.cc:374] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:01.862721 24462 raft_consensus.cc:729] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b833b0efbaa403f93803735123f5fed, State: Initialized, Role: FOLLOWER
I20260619 22:30:01.863217 24462 consensus_queue.cc:260] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:01.864248 24462 ts_tablet_manager.cc:1428] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:30:01.864548 24444 heartbeater.cc:503] Master 127.5.176.254:33797 was elected leader, sending a full tablet report...
W20260619 22:30:02.026306 24445 tablet.cc:2369] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:30:02.057664 24311 tablet.cc:2369] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:30:02.092767 24467 raft_consensus.cc:482] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:30:02.093043 24467 raft_consensus.cc:504] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:02.094435 24467 leader_election.cc:288] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b833b0efbaa403f93803735123f5fed (127.5.176.195:39605), b25680dd0965488191d8536392255f0c (127.5.176.193:34597)
W20260619 22:30:02.096614 24177 tablet.cc:2369] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:30:02.099272 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.095386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:46797 (local address 127.5.176.193:34597)
0619 22:30:02.095524 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:02.095529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:02.095543 (+ 14us) server_negotiation.cc:407] Connection header received
0619 22:30:02.095702 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:02.095705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:02.095758 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:02.095859 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:02.096117 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.096732 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.098196 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.098607 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.098764 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.098816 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:02.099068 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 22:30:02.099142 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:30:02.099280 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.094864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:37063 (local address 127.5.176.195:39605)
0619 22:30:02.095069 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:02.095074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:02.095257 (+ 183us) server_negotiation.cc:407] Connection header received
0619 22:30:02.095439 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:02.095446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:02.095498 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:02.095550 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:02.095937 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.096486 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.098171 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.098589 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.098749 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.098812 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:02.099039 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 22:30:02.099128 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 22:30:02.099293 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.095180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34597 (local address 127.5.176.194:46797)
0619 22:30:02.095453 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:02.095468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:02.095574 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:02.095873 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:02.095875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:02.095891 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:02.096011 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.096016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.096851 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.096855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.098024 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.098030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.098774 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.098778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.098905 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.098923 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 22:30:02.099030 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 22:30:02.099143 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":53,"thread_start_us":69,"threads_started":1}
I20260619 22:30:02.099301 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.094773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:39605 (local address 127.5.176.194:37063)
0619 22:30:02.095118 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:02.095135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:02.095284 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:02.095678 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:02.095680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:02.095695 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:02.095823 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.095828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.096620 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.096625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.098003 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.098014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.098726 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.098730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.098880 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.098902 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:30:02.098972 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 22:30:02.099097 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":7,"thread_start_us":145,"threads_started":1}
I20260619 22:30:02.100111 24130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f60fb23f14c47e38dda9e629fa65f3a" candidate_uuid: "132d8d2f4640463d9b2acdccc94ded44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25680dd0965488191d8536392255f0c" is_pre_election: true
I20260619 22:30:02.100157 24398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f60fb23f14c47e38dda9e629fa65f3a" candidate_uuid: "132d8d2f4640463d9b2acdccc94ded44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b833b0efbaa403f93803735123f5fed" is_pre_election: true
I20260619 22:30:02.100459 24130 raft_consensus.cc:2454] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 132d8d2f4640463d9b2acdccc94ded44 in term 0.
I20260619 22:30:02.100459 24398 raft_consensus.cc:2454] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 132d8d2f4640463d9b2acdccc94ded44 in term 0.
I20260619 22:30:02.100984 24197 leader_election.cc:302] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132d8d2f4640463d9b2acdccc94ded44, b25680dd0965488191d8536392255f0c; no voters:
I20260619 22:30:02.101274 24467 raft_consensus.cc:2781] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:30:02.101356 24467 raft_consensus.cc:482] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:30:02.101389 24467 raft_consensus.cc:3037] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:02.102506 24467 raft_consensus.cc:504] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:02.102856 24467 leader_election.cc:288] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [CANDIDATE]: Term 1 election: Requested vote from peers 7b833b0efbaa403f93803735123f5fed (127.5.176.195:39605), b25680dd0965488191d8536392255f0c (127.5.176.193:34597)
I20260619 22:30:02.103241 24398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f60fb23f14c47e38dda9e629fa65f3a" candidate_uuid: "132d8d2f4640463d9b2acdccc94ded44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b833b0efbaa403f93803735123f5fed"
I20260619 22:30:02.103334 24130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f60fb23f14c47e38dda9e629fa65f3a" candidate_uuid: "132d8d2f4640463d9b2acdccc94ded44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25680dd0965488191d8536392255f0c"
I20260619 22:30:02.103386 24398 raft_consensus.cc:3037] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:02.103575 24130 raft_consensus.cc:3037] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:02.104656 24398 raft_consensus.cc:2454] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 132d8d2f4640463d9b2acdccc94ded44 in term 1.
I20260619 22:30:02.104666 24130 raft_consensus.cc:2454] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 132d8d2f4640463d9b2acdccc94ded44 in term 1.
I20260619 22:30:02.105033 24197 leader_election.cc:302] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132d8d2f4640463d9b2acdccc94ded44, 7b833b0efbaa403f93803735123f5fed; no voters:
I20260619 22:30:02.105289 24467 raft_consensus.cc:2781] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:02.105574 24467 raft_consensus.cc:686] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 1 LEADER]: Becoming Leader. State: Replica: 132d8d2f4640463d9b2acdccc94ded44, State: Running, Role: LEADER
I20260619 22:30:02.105962 24467 consensus_queue.cc:237] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } }
I20260619 22:30:02.109292 23985 catalog_manager.cc:5495] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 132d8d2f4640463d9b2acdccc94ded44 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "132d8d2f4640463d9b2acdccc94ded44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132d8d2f4640463d9b2acdccc94ded44" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 34121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20260619 22:30:02.131974 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.128058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.5.176.194:34121)
0619 22:30:02.128189 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:02.128193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:02.128316 (+ 123us) server_negotiation.cc:407] Connection header received
0619 22:30:02.128524 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:02.128526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:02.128583 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:02.128697 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:02.128960 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.129476 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.130429 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.130648 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.130759 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.131111 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:02.131530 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:02.131690 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:02.131778 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 22:30:02.131827 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 22:30:02.138564 24398 raft_consensus.cc:1261] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 1 FOLLOWER]: Refusing update from remote peer 132d8d2f4640463d9b2acdccc94ded44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:02.138564 24130 raft_consensus.cc:1261] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 1 FOLLOWER]: Refusing update from remote peer 132d8d2f4640463d9b2acdccc94ded44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:02.139401 24467 consensus_queue.cc:1035] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25680dd0965488191d8536392255f0c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:02.139606 24472 consensus_queue.cc:1035] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b833b0efbaa403f93803735123f5fed" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:02.148569 24476 mvcc.cc:205] Tried to move back new op lower bound from 7298695995951112192 to 7298695995829178368. Current Snapshot: MvccSnapshot[applied={T|T < 7298695995951112192}]
I20260619 22:30:02.149566 24477 mvcc.cc:205] Tried to move back new op lower bound from 7298695995951112192 to 7298695995829178368. Current Snapshot: MvccSnapshot[applied={T|T < 7298695995951112192}]
I20260619 22:30:02.158879 23984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:02.159094 23984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:02.161473 23984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35d275c98c614755b6c0a54984c733db: Sending DeleteTablet for 3 replicas of tablet 1f60fb23f14c47e38dda9e629fa65f3a
I20260619 22:30:02.162290 24244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f60fb23f14c47e38dda9e629fa65f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:02 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:59162
I20260619 22:30:02.162415 24110 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f60fb23f14c47e38dda9e629fa65f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:02 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:36688
I20260619 22:30:02.162628 24378 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f60fb23f14c47e38dda9e629fa65f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:02 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:45310
I20260619 22:30:02.162930 24486 tablet_replica.cc:330] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c: stopping tablet replica
I20260619 22:30:02.163125 24485 tablet_replica.cc:330] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: stopping tablet replica
I20260619 22:30:02.163144 24487 tablet_replica.cc:330] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: stopping tablet replica
I20260619 22:30:02.163453 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24047
I20260619 22:30:02.163522 24487 raft_consensus.cc:2229] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:02.163853 24486 raft_consensus.cc:2229] T 1f60fb23f14c47e38dda9e629fa65f3a P b25680dd0965488191d8536392255f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:02.163887 24487 raft_consensus.cc:2258] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:02.164104 24485 raft_consensus.cc:2229] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:30:02.164896 24485 raft_consensus.cc:2258] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:02.166985 24485 ts_tablet_manager.cc:1905] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:02.167939 24487 ts_tablet_manager.cc:1905] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:02.170758 24487 ts_tablet_manager.cc:1918] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:02.170881 24487 log.cc:1196] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/1f60fb23f14c47e38dda9e629fa65f3a
I20260619 22:30:02.171216 24487 ts_tablet_manager.cc:1939] T 1f60fb23f14c47e38dda9e629fa65f3a P 7b833b0efbaa403f93803735123f5fed: Deleting consensus metadata
I20260619 22:30:02.171203 24485 ts_tablet_manager.cc:1918] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:02.171458 24485 log.cc:1196] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/1f60fb23f14c47e38dda9e629fa65f3a
I20260619 22:30:02.171831 24485 ts_tablet_manager.cc:1939] T 1f60fb23f14c47e38dda9e629fa65f3a P 132d8d2f4640463d9b2acdccc94ded44: Deleting consensus metadata
I20260619 22:30:02.172657 23969 catalog_manager.cc:4841] TS 7b833b0efbaa403f93803735123f5fed (127.5.176.195:39605): tablet 1f60fb23f14c47e38dda9e629fa65f3a (table test-table [id=669c3dc48a804a82bf68231944615233]) successfully deleted
I20260619 22:30:02.173084 23969 catalog_manager.cc:4841] TS 132d8d2f4640463d9b2acdccc94ded44 (127.5.176.194:34121): tablet 1f60fb23f14c47e38dda9e629fa65f3a (table test-table [id=669c3dc48a804a82bf68231944615233]) successfully deleted
W20260619 22:30:02.173342 23969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:34597 (error 108)
I20260619 22:30:02.173732 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24181
I20260619 22:30:02.175222 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.174928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:34597 (local address 127.0.0.1:36690)
0619 22:30:02.175075 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:02.175146 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 22:30:02.175411 23969 catalog_manager.cc:4584] TS b25680dd0965488191d8536392255f0c (127.5.176.193:34597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f60fb23f14c47e38dda9e629fa65f3a: Network error: Client connection negotiation failed: client connection to 127.5.176.193:34597: connect: Connection refused (error 111)
I20260619 22:30:02.180097 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24315
I20260619 22:30:02.185961 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 23953
2026-06-19T22:30:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2485 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T22:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:02Z Disabled control of system clock
I20260619 22:30:02.217392 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38481
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:38481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:30:02.330018 24492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:02.330376 24492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:02.330415 24492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:02.334878 24492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:02.334975 24492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:02.334993 24492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:02.335012 24492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:02.335029 24492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:02.340634 24492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:38481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:02.341900 24492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:02.343429 24492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:02.352327 24497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:02.352341 24498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:02.352411 24492 server_base.cc:1034] running on GCE node
W20260619 22:30:02.352330 24500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:02.353211 24492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:02.354421 24492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:02.355592 24492 hybrid_clock.cc:648] HybridClock initialized: now 1781908202355593 us; error 44 us; skew 500 ppm
I20260619 22:30:02.357415 24492 webserver.cc:460] Webserver started at http://127.5.176.254:43185/ using document root <none> and password file <none>
I20260619 22:30:02.357967 24492 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:02.358027 24492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:02.358266 24492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:02.360298 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "a1a5e5024b2548379febd78ffbfbb626"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "45509c85ba3caf01cd839bc4ed89c8e7"
server_key_iv: "ba1d523e4a81312d9e8c8ae86cbb0f68"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.360800 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "a1a5e5024b2548379febd78ffbfbb626"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "45509c85ba3caf01cd839bc4ed89c8e7"
server_key_iv: "ba1d523e4a81312d9e8c8ae86cbb0f68"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.364526 24492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:02.367048 24506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.368413 24492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:02.368536 24492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "a1a5e5024b2548379febd78ffbfbb626"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "45509c85ba3caf01cd839bc4ed89c8e7"
server_key_iv: "ba1d523e4a81312d9e8c8ae86cbb0f68"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.368633 24492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:02.387392 24492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:02.388259 24492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:02.388468 24492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:02.395946 24492 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:38481
I20260619 22:30:02.395967 24558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:38481 every 8 connection(s)
I20260619 22:30:02.396957 24492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:02.400218 24559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:02.403858 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24492
I20260619 22:30:02.404012 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:02.404302 5827 external_mini_cluster.cc:1428] Setting key 6f7ab6af9016852be7a9b1eec7a3e2cd
I20260619 22:30:02.406780 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626: Bootstrap starting.
I20260619 22:30:02.409050 24559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:02.409850 24559 log.cc:824] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:02.411834 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626: No bootstrap required, opened a new log
I20260619 22:30:02.414122 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.406004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.5.176.254:38481)
0619 22:30:02.406757 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:02.406773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:02.406809 (+ 36us) server_negotiation.cc:407] Connection header received
0619 22:30:02.407524 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:02.407540 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:02.407924 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:02.408259 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:02.408706 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.409737 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.410941 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.411411 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.411585 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.412470 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:02.412475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:02.412488 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:02.412862 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:02.412924 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:02.412943 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:02.413051 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:02.413173 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 22:30:02.413436 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":149,"threads_started":1}
I20260619 22:30:02.415807 24559 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a5e5024b2548379febd78ffbfbb626" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38481 } }
I20260619 22:30:02.416075 24559 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:02.416124 24559 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a5e5024b2548379febd78ffbfbb626, State: Initialized, Role: FOLLOWER
I20260619 22:30:02.416606 24559 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a5e5024b2548379febd78ffbfbb626" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38481 } }
I20260619 22:30:02.416767 24559 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:02.416814 24559 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:02.416886 24559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:02.417841 24559 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a5e5024b2548379febd78ffbfbb626" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38481 } }
I20260619 22:30:02.418259 24559 leader_election.cc:302] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1a5e5024b2548379febd78ffbfbb626; no voters:
I20260619 22:30:02.418648 24559 leader_election.cc:288] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:02.418833 24564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:02.419106 24564 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [term 1 LEADER]: Becoming Leader. State: Replica: a1a5e5024b2548379febd78ffbfbb626, State: Running, Role: LEADER
I20260619 22:30:02.419436 24564 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a5e5024b2548379febd78ffbfbb626" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38481 } }
I20260619 22:30:02.419938 24559 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:02.421281 24565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1a5e5024b2548379febd78ffbfbb626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a5e5024b2548379febd78ffbfbb626" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38481 } } }
I20260619 22:30:02.421419 24565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:02.421797 24573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:02.421766 24566 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1a5e5024b2548379febd78ffbfbb626. Latest consensus state: current_term: 1 leader_uuid: "a1a5e5024b2548379febd78ffbfbb626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a5e5024b2548379febd78ffbfbb626" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38481 } } }
I20260619 22:30:02.421871 24566 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:02.425217 24573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:02.431283 24573 catalog_manager.cc:1347] Generated new cluster ID: f9cb57c46aac4c258471c29ee6254761
I20260619 22:30:02.431370 24573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:02.446185 24573 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:02.447458 24573 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:02.458352 24573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626: Generated new TSK 0
I20260619 22:30:02.459275 24573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:02.472584 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38481
--builtin_ntp_servers=127.5.176.212:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:30:02.590446 24583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:02.591064 24583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:02.591194 24583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:02.595700 24583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:02.595860 24583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:02.595978 24583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:30:02.601289 24583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:02.602528 24583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:02.603925 24583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:02.613831 24591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:02.613842 24588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:02.613873 24589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:02.613916 24583 server_base.cc:1034] running on GCE node
I20260619 22:30:02.614486 24583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:02.615171 24583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:02.616356 24583 hybrid_clock.cc:648] HybridClock initialized: now 1781908202616340 us; error 27 us; skew 500 ppm
I20260619 22:30:02.618340 24583 webserver.cc:460] Webserver started at http://127.5.176.193:37087/ using document root <none> and password file <none>
I20260619 22:30:02.619012 24583 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:02.619089 24583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:02.619333 24583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:02.621800 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "f56a20f28b044e9a93309303ceffd816"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "c6b6e7835fa08ecc83eabbe87530bf39"
server_key_iv: "294579cac57bfca1dc015f9a64bbe0c8"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.622412 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "f56a20f28b044e9a93309303ceffd816"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "c6b6e7835fa08ecc83eabbe87530bf39"
server_key_iv: "294579cac57bfca1dc015f9a64bbe0c8"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.627669 24583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 22:30:02.631008 24597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.632627 24583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 22:30:02.632761 24583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "f56a20f28b044e9a93309303ceffd816"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "c6b6e7835fa08ecc83eabbe87530bf39"
server_key_iv: "294579cac57bfca1dc015f9a64bbe0c8"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.632892 24583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:02.653888 24583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:02.654676 24583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:02.654888 24583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:02.655509 24583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:02.656548 24583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:02.656594 24583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.656663 24583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:02.656693 24583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.665956 24583 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:46095
I20260619 22:30:02.665974 24710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:46095 every 8 connection(s)
I20260619 22:30:02.666982 24583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:30:02.669942 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24583
I20260619 22:30:02.670074 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:30:02.670379 5827 external_mini_cluster.cc:1428] Setting key ec9ccda9758aa4e6a9c091c25f1a9513
I20260619 22:30:02.672570 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38481
--builtin_ntp_servers=127.5.176.212:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:30:02.678463 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.668864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:48765 (local address 127.5.176.254:38481)
0619 22:30:02.669042 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:02.669045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:02.670052 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 22:30:02.671877 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:02.671880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:02.671934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:02.672018 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:02.673206 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.674257 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.676027 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.676437 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.676579 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.677780 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:02.677786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:02.677791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:02.677822 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:02.677850 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:02.677865 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:02.677958 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:02.678110 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 22:30:02.678238 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 22:30:02.679320 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.669230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38481 (local address 127.5.176.193:48765)
0619 22:30:02.669892 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:02.669933 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:02.671499 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:02.672261 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:02.672287 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:02.672397 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:02.673039 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.673054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.674430 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.674436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.675835 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.675847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.676615 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.676618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.676771 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.677478 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:30:02.677479 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:02.677525 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:30:02.677598 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:02.677962 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:02.677969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:02.677974 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 22:30:02.678077 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 22:30:02.678400 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260619 22:30:02.680819 24711 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38481
I20260619 22:30:02.681080 24711 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:02.681648 24711 heartbeater.cc:511] Master 127.5.176.254:38481 requested a full tablet report, sending...
I20260619 22:30:02.683241 24523 ts_manager.cc:194] Registered new tserver with Master: f56a20f28b044e9a93309303ceffd816 (127.5.176.193:46095)
I20260619 22:30:02.684418 24523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.193:48765
W20260619 22:30:02.789276 24714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:02.789604 24714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:02.789680 24714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:02.793892 24714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:02.793963 24714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:02.794021 24714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:30:02.798923 24714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:02.800087 24714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:02.801396 24714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:02.811362 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:02.811362 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:02.811363 24722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:02.811779 24714 server_base.cc:1034] running on GCE node
I20260619 22:30:02.812278 24714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:02.813057 24714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:02.814217 24714 hybrid_clock.cc:648] HybridClock initialized: now 1781908202814203 us; error 35 us; skew 500 ppm
I20260619 22:30:02.816316 24714 webserver.cc:460] Webserver started at http://127.5.176.194:36455/ using document root <none> and password file <none>
I20260619 22:30:02.816947 24714 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:02.817054 24714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:02.817291 24714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:02.819532 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "c6b58b2cfbd741ff9171d66174d802d7"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "2cd00eae61943a73fdc6a18a0682e76d"
server_key_iv: "f44921f817a23451e94678868ddbac2e"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.820163 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b58b2cfbd741ff9171d66174d802d7"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "2cd00eae61943a73fdc6a18a0682e76d"
server_key_iv: "f44921f817a23451e94678868ddbac2e"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.824337 24714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:02.827167 24728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.828624 24714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:02.828789 24714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "c6b58b2cfbd741ff9171d66174d802d7"
format_stamp: "Formatted at 2026-06-19 22:30:02 on dist-test-slave-bt27"
server_key: "2cd00eae61943a73fdc6a18a0682e76d"
server_key_iv: "f44921f817a23451e94678868ddbac2e"
server_key_version: "encryptionkey@0"
I20260619 22:30:02.828931 24714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:02.855158 24714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:02.855926 24714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:02.856139 24714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:02.856742 24714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:02.857764 24714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:02.857836 24714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.857910 24714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:02.857967 24714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:02.867394 24714 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:41609
I20260619 22:30:02.867409 24841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:41609 every 8 connection(s)
I20260619 22:30:02.868368 24714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:30:02.870777 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24714
I20260619 22:30:02.870908 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:30:02.871261 5827 external_mini_cluster.cc:1428] Setting key 06fa24844bbe1059d7ec8ba02ca8cd47
I20260619 22:30:02.873354 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38481
--builtin_ntp_servers=127.5.176.212:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:30:02.880213 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.870304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:35415 (local address 127.5.176.254:38481)
0619 22:30:02.870470 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:02.870474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:02.871810 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 22:30:02.873449 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:02.873452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:02.873517 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:02.873608 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:02.875038 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.875933 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.877804 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.878046 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.878259 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.879697 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:02.879700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:02.879702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:02.879716 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:02.879736 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:02.879745 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:02.879812 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:02.880031 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 22:30:02.880087 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:30:02.881573 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:02.870808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38481 (local address 127.5.176.194:35415)
0619 22:30:02.871596 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:02.871666 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:02.873230 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:02.873967 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:02.873979 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:02.874098 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:02.874843 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.874880 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.876110 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.876117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.877648 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:02.877661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:02.878319 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:02.878326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:02.878577 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:02.879335 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:30:02.879339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:02.879405 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:30:02.879506 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:02.879842 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:02.879854 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:02.879863 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:30:02.880025 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 22:30:02.880384 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":148,"threads_started":1}
I20260619 22:30:02.882963 24842 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38481
I20260619 22:30:02.883216 24842 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:02.883785 24842 heartbeater.cc:511] Master 127.5.176.254:38481 requested a full tablet report, sending...
I20260619 22:30:02.884900 24523 ts_manager.cc:194] Registered new tserver with Master: c6b58b2cfbd741ff9171d66174d802d7 (127.5.176.194:41609)
I20260619 22:30:02.885490 24523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.194:35415
W20260619 22:30:02.987584 24845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:02.987936 24845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:02.988009 24845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:02.992301 24845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:02.992408 24845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:02.992501 24845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:30:02.997421 24845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:38481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:02.998629 24845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:03.000052 24845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:30:03.010334 24845 server_base.cc:1034] running on GCE node
W20260619 22:30:03.010283 24851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:03.010280 24853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:03.010423 24850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:03.011183 24845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:03.011862 24845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:03.013054 24845 hybrid_clock.cc:648] HybridClock initialized: now 1781908203013037 us; error 36 us; skew 500 ppm
I20260619 22:30:03.015076 24845 webserver.cc:460] Webserver started at http://127.5.176.195:39597/ using document root <none> and password file <none>
I20260619 22:30:03.015686 24845 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:03.015784 24845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:03.016028 24845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:03.018078 24845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "83919f23bdc943f7859dfa6c681a7a12"
format_stamp: "Formatted at 2026-06-19 22:30:03 on dist-test-slave-bt27"
server_key: "a7c741498463bf5c74065a665bda7b83"
server_key_iv: "4cc47643f9c806797627ab0f305c32dc"
server_key_version: "encryptionkey@0"
I20260619 22:30:03.018697 24845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "83919f23bdc943f7859dfa6c681a7a12"
format_stamp: "Formatted at 2026-06-19 22:30:03 on dist-test-slave-bt27"
server_key: "a7c741498463bf5c74065a665bda7b83"
server_key_iv: "4cc47643f9c806797627ab0f305c32dc"
server_key_version: "encryptionkey@0"
I20260619 22:30:03.022740 24845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:30:03.025161 24859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:03.026446 24845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:03.026576 24845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "83919f23bdc943f7859dfa6c681a7a12"
format_stamp: "Formatted at 2026-06-19 22:30:03 on dist-test-slave-bt27"
server_key: "a7c741498463bf5c74065a665bda7b83"
server_key_iv: "4cc47643f9c806797627ab0f305c32dc"
server_key_version: "encryptionkey@0"
I20260619 22:30:03.026692 24845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:03.040725 24845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:03.041407 24845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:03.041605 24845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:03.042255 24845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:03.043208 24845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:03.043282 24845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:03.043359 24845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:03.043401 24845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:03.052119 24845 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:45357
I20260619 22:30:03.052141 24972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:45357 every 8 connection(s)
I20260619 22:30:03.053076 24845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:30:03.060945 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 24845
I20260619 22:30:03.061105 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:30:03.061465 5827 external_mini_cluster.cc:1428] Setting key 8ded6b63ae4995765e2c704c71f051a9
I20260619 22:30:03.062615 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.055003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:38517 (local address 127.5.176.254:38481)
0619 22:30:03.055151 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.055155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.056195 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 22:30:03.057542 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.057545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.057593 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.057672 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:03.058774 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.059302 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.060447 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.060729 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.060902 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.062064 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:03.062067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:03.062070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:03.062082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:03.062105 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:03.062115 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:03.062217 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.062403 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.062466 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 22:30:03.063630 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.055433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:38481 (local address 127.5.176.195:38517)
0619 22:30:03.056069 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.056109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:03.057344 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:03.057918 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:03.057925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:03.058010 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:03.058620 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.058634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.059433 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.059436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.060314 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.060323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.060905 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.060908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.061048 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.061793 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:30:03.061825 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:03.061867 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:30:03.061933 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:03.062227 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:03.062235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:03.062241 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:30:03.062389 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 22:30:03.062715 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260619 22:30:03.064754 24973 heartbeater.cc:347] Connected to a master server at 127.5.176.254:38481
I20260619 22:30:03.065049 24973 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:03.065539 24973 heartbeater.cc:511] Master 127.5.176.254:38481 requested a full tablet report, sending...
I20260619 22:30:03.066584 24523 ts_manager.cc:194] Registered new tserver with Master: 83919f23bdc943f7859dfa6c681a7a12 (127.5.176.195:45357)
I20260619 22:30:03.067226 24523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.195:38517
I20260619 22:30:03.075459 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:30:03.081086 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.077364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.5.176.254:38481)
0619 22:30:03.077509 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.077514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.077687 (+ 173us) server_negotiation.cc:407] Connection header received
0619 22:30:03.077785 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.077787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.077828 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.077899 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:03.078195 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.078712 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.079495 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.079692 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.079764 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.080599 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:03.080602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:03.080604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:03.080614 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:03.080629 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:03.080636 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:03.080721 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.080883 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.080926 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:30:03.084024 24523 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:30:03.087105 24523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:30:03.101970 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.096857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.5.176.194:41609)
0619 22:30:03.097225 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.097239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.097263 (+ 24us) server_negotiation.cc:407] Connection header received
0619 22:30:03.097337 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.097343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.097545 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.097697 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:03.098110 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.099285 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.100687 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.101272 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.101402 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.101610 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.101701 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.101798 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260619 22:30:03.101965 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.096593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:41609 (local address 127.0.0.1:45580)
0619 22:30:03.096891 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.096905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:03.096995 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:03.097679 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:03.097682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:03.097699 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:03.097972 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.097980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.099448 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.099452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.100554 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.100564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.101401 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.101404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.101521 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.101561 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 22:30:03.101674 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 22:30:03.101798 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":63,"thread_start_us":107,"threads_started":1}
I20260619 22:30:03.102240 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.095955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:46095 (local address 127.0.0.1:59062)
0619 22:30:03.096353 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.096392 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:03.096505 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:03.097689 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:03.097692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:03.097711 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:03.097986 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.097993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.099405 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.099408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.100640 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.100652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.101689 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.101692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.101831 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.101882 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 22:30:03.101949 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 22:30:03.102047 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":244,"spinlock_wait_cycles":6528,"thread_start_us":125,"threads_started":1}
I20260619 22:30:03.102561 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.095894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.5.176.195:45357)
0619 22:30:03.096991 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.097008 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.097034 (+ 26us) server_negotiation.cc:407] Connection header received
0619 22:30:03.097115 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.097121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.097326 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.097471 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:03.098075 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.098912 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.100540 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.101346 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.101453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.101636 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.102301 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.102397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":189,"threads_started":1}
I20260619 22:30:03.102241 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.095695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:45357 (local address 127.0.0.1:39630)
0619 22:30:03.096323 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.096372 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:03.096503 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:03.097653 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:03.097661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:03.097688 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:03.097932 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.097940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.099041 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.099045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.100411 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.100421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.101689 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.101692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.101833 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.101884 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 22:30:03.101961 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 22:30:03.102047 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":190,"thread_start_us":118,"threads_started":1}
I20260619 22:30:03.103199 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.096813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.5.176.193:46095)
0619 22:30:03.097208 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.097219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.097237 (+ 18us) server_negotiation.cc:407] Connection header received
0619 22:30:03.097303 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.097308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.097547 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.097877 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:03.098110 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.099285 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.100768 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.101554 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.102327 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.102516 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.102779 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.102885 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":97,"threads_started":1}
I20260619 22:30:03.105574 24907 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb89b94629a462f860179fe5162bb9d (DEFAULT_TABLE table=test-table [id=26208077dbfd45e88c48636cdd03a2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:03.105700 24645 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb89b94629a462f860179fe5162bb9d (DEFAULT_TABLE table=test-table [id=26208077dbfd45e88c48636cdd03a2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:03.105701 24776 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb89b94629a462f860179fe5162bb9d (DEFAULT_TABLE table=test-table [id=26208077dbfd45e88c48636cdd03a2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:03.106657 24907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb89b94629a462f860179fe5162bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:03.106729 24776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb89b94629a462f860179fe5162bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:03.106729 24645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb89b94629a462f860179fe5162bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:03.113359 24990 tablet_bootstrap.cc:492] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Bootstrap starting.
I20260619 22:30:03.114619 24991 tablet_bootstrap.cc:492] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: Bootstrap starting.
I20260619 22:30:03.115751 24990 tablet_bootstrap.cc:654] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:03.116200 24992 tablet_bootstrap.cc:492] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Bootstrap starting.
I20260619 22:30:03.116470 24991 tablet_bootstrap.cc:654] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:03.116669 24990 log.cc:824] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:03.117281 24991 log.cc:824] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:03.118288 24992 tablet_bootstrap.cc:654] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:03.118928 24992 log.cc:824] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:03.119270 24991 tablet_bootstrap.cc:492] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: No bootstrap required, opened a new log
I20260619 22:30:03.119344 24990 tablet_bootstrap.cc:492] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: No bootstrap required, opened a new log
I20260619 22:30:03.119490 24991 ts_tablet_manager.cc:1397] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:30:03.119493 24990 ts_tablet_manager.cc:1397] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 22:30:03.120932 24992 tablet_bootstrap.cc:492] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: No bootstrap required, opened a new log
I20260619 22:30:03.121146 24992 ts_tablet_manager.cc:1397] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 22:30:03.124060 24991 raft_consensus.cc:348] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.124060 24990 raft_consensus.cc:348] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.124356 24991 raft_consensus.cc:374] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:03.124356 24990 raft_consensus.cc:374] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:03.124408 24991 raft_consensus.cc:729] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56a20f28b044e9a93309303ceffd816, State: Initialized, Role: FOLLOWER
I20260619 22:30:03.124408 24990 raft_consensus.cc:729] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83919f23bdc943f7859dfa6c681a7a12, State: Initialized, Role: FOLLOWER
I20260619 22:30:03.124940 24991 consensus_queue.cc:260] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.124929 24990 consensus_queue.cc:260] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.125289 24992 raft_consensus.cc:348] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.125563 24992 raft_consensus.cc:374] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:03.125627 24992 raft_consensus.cc:729] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b58b2cfbd741ff9171d66174d802d7, State: Initialized, Role: FOLLOWER
I20260619 22:30:03.126015 24990 ts_tablet_manager.cc:1428] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:30:03.126015 24991 ts_tablet_manager.cc:1428] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:30:03.126078 24992 consensus_queue.cc:260] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.126421 24973 heartbeater.cc:503] Master 127.5.176.254:38481 was elected leader, sending a full tablet report...
I20260619 22:30:03.126535 24711 heartbeater.cc:503] Master 127.5.176.254:38481 was elected leader, sending a full tablet report...
I20260619 22:30:03.128264 24992 ts_tablet_manager.cc:1428] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:30:03.128614 24842 heartbeater.cc:503] Master 127.5.176.254:38481 was elected leader, sending a full tablet report...
W20260619 22:30:03.168277 24712 tablet.cc:2369] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 22:30:03.304143 24974 tablet.cc:2369] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:30:03.352047 24997 raft_consensus.cc:482] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:30:03.352298 24997 raft_consensus.cc:504] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.353806 24997 leader_election.cc:288] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f56a20f28b044e9a93309303ceffd816 (127.5.176.193:46095), c6b58b2cfbd741ff9171d66174d802d7 (127.5.176.194:41609)
I20260619 22:30:03.357476 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.354065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:41609 (local address 127.5.176.195:40589)
0619 22:30:03.354307 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.354323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:03.354453 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:03.354710 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:03.354713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:03.354726 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:03.354871 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.354877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.355599 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.355606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.356394 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.356403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.357036 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.357039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.357135 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.357157 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 22:30:03.357253 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 22:30:03.357340 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":6}
I20260619 22:30:03.357476 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.354229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:40589 (local address 127.5.176.194:41609)
0619 22:30:03.354353 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.354359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.354397 (+ 38us) server_negotiation.cc:407] Connection header received
0619 22:30:03.354567 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.354570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.354631 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.354705 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:03.354988 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.355488 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.356525 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.356914 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.357043 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.357096 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.357268 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.357350 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 22:30:03.357486 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.354163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:56351 (local address 127.5.176.193:46095)
0619 22:30:03.354605 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.354608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.354620 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:30:03.354666 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.354668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.354705 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.354791 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:03.355074 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.355562 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.356559 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.356940 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.357065 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.357110 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.357289 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.357363 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260619 22:30:03.357501 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.354070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:46095 (local address 127.5.176.195:56351)
0619 22:30:03.354464 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.354477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:03.354580 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:03.354813 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:03.354816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:03.354839 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:03.354961 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.354966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.355668 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.355673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.356440 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:03.356447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.357065 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.357069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:03.357173 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.357189 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 22:30:03.357274 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 22:30:03.357355 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":37,"thread_start_us":107,"threads_started":1}
I20260619 22:30:03.358242 24796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb89b94629a462f860179fe5162bb9d" candidate_uuid: "83919f23bdc943f7859dfa6c681a7a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b58b2cfbd741ff9171d66174d802d7" is_pre_election: true
I20260619 22:30:03.358332 24665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb89b94629a462f860179fe5162bb9d" candidate_uuid: "83919f23bdc943f7859dfa6c681a7a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56a20f28b044e9a93309303ceffd816" is_pre_election: true
I20260619 22:30:03.358595 24796 raft_consensus.cc:2454] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83919f23bdc943f7859dfa6c681a7a12 in term 0.
I20260619 22:30:03.358639 24665 raft_consensus.cc:2454] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83919f23bdc943f7859dfa6c681a7a12 in term 0.
I20260619 22:30:03.359120 24860 leader_election.cc:302] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83919f23bdc943f7859dfa6c681a7a12, c6b58b2cfbd741ff9171d66174d802d7; no voters:
I20260619 22:30:03.359380 24997 raft_consensus.cc:2781] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:30:03.359463 24997 raft_consensus.cc:482] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:30:03.359505 24997 raft_consensus.cc:3037] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:03.360519 24997 raft_consensus.cc:504] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.360847 24997 leader_election.cc:288] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [CANDIDATE]: Term 1 election: Requested vote from peers f56a20f28b044e9a93309303ceffd816 (127.5.176.193:46095), c6b58b2cfbd741ff9171d66174d802d7 (127.5.176.194:41609)
I20260619 22:30:03.361161 24665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb89b94629a462f860179fe5162bb9d" candidate_uuid: "83919f23bdc943f7859dfa6c681a7a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56a20f28b044e9a93309303ceffd816"
I20260619 22:30:03.361197 24796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb89b94629a462f860179fe5162bb9d" candidate_uuid: "83919f23bdc943f7859dfa6c681a7a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b58b2cfbd741ff9171d66174d802d7"
I20260619 22:30:03.361325 24796 raft_consensus.cc:3037] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:03.361296 24665 raft_consensus.cc:3037] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:03.362401 24665 raft_consensus.cc:2454] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83919f23bdc943f7859dfa6c681a7a12 in term 1.
I20260619 22:30:03.362439 24796 raft_consensus.cc:2454] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83919f23bdc943f7859dfa6c681a7a12 in term 1.
I20260619 22:30:03.362751 24861 leader_election.cc:302] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83919f23bdc943f7859dfa6c681a7a12, f56a20f28b044e9a93309303ceffd816; no voters:
I20260619 22:30:03.363020 24997 raft_consensus.cc:2781] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:03.363246 24997 raft_consensus.cc:686] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 1 LEADER]: Becoming Leader. State: Replica: 83919f23bdc943f7859dfa6c681a7a12, State: Running, Role: LEADER
I20260619 22:30:03.363595 24997 consensus_queue.cc:237] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } }
I20260619 22:30:03.366871 24522 catalog_manager.cc:5495] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83919f23bdc943f7859dfa6c681a7a12 (127.5.176.195). New cstate: current_term: 1 leader_uuid: "83919f23bdc943f7859dfa6c681a7a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83919f23bdc943f7859dfa6c681a7a12" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 45357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 } health_report { overall_health: UNKNOWN } } }
W20260619 22:30:03.369527 24843 tablet.cc:2369] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:30:03.400700 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.396989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.5.176.195:45357)
0619 22:30:03.397130 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:03.397136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:03.397151 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:30:03.397273 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:03.397276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:03.397345 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:03.397465 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:03.397717 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.398323 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.399178 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:03.399418 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:03.399528 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:03.399843 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:03.400216 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:03.400380 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:03.400496 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 22:30:03.400532 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 22:30:03.407212 24665 raft_consensus.cc:1261] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 1 FOLLOWER]: Refusing update from remote peer 83919f23bdc943f7859dfa6c681a7a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:03.407241 24796 raft_consensus.cc:1261] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 1 FOLLOWER]: Refusing update from remote peer 83919f23bdc943f7859dfa6c681a7a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:03.407975 24997 consensus_queue.cc:1035] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b58b2cfbd741ff9171d66174d802d7" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:03.408210 25000 consensus_queue.cc:1035] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56a20f28b044e9a93309303ceffd816" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:03.416400 25005 mvcc.cc:205] Tried to move back new op lower bound from 7298696001147834368 to 7298696000980185088. Current Snapshot: MvccSnapshot[applied={T|T < 7298696001147834368}]
I20260619 22:30:03.418183 25004 mvcc.cc:205] Tried to move back new op lower bound from 7298696001147834368 to 7298696000980185088. Current Snapshot: MvccSnapshot[applied={T|T < 7298696001147834368}]
I20260619 22:30:03.423533 24522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:03.423722 24522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:03.426506 24522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1a5e5024b2548379febd78ffbfbb626: Sending DeleteTablet for 3 replicas of tablet 2cb89b94629a462f860179fe5162bb9d
I20260619 22:30:03.427393 24907 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb89b94629a462f860179fe5162bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:03 UTC) from {username='slave'} at 127.0.0.1:39630
I20260619 22:30:03.427507 24645 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb89b94629a462f860179fe5162bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:03 UTC) from {username='slave'} at 127.0.0.1:59062
I20260619 22:30:03.427526 24776 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb89b94629a462f860179fe5162bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:03 UTC) from {username='slave'} at 127.0.0.1:45580
I20260619 22:30:03.428062 25014 tablet_replica.cc:330] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816: stopping tablet replica
I20260619 22:30:03.428256 25013 tablet_replica.cc:330] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: stopping tablet replica
I20260619 22:30:03.428516 25014 raft_consensus.cc:2229] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:03.428565 25013 raft_consensus.cc:2229] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:30:03.428946 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24583
I20260619 22:30:03.429003 25013 raft_consensus.cc:2258] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:03.428969 25014 raft_consensus.cc:2258] T 2cb89b94629a462f860179fe5162bb9d P f56a20f28b044e9a93309303ceffd816 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:03.429466 25015 tablet_replica.cc:330] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: stopping tablet replica
I20260619 22:30:03.429807 25015 raft_consensus.cc:2229] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:03.430183 25015 raft_consensus.cc:2258] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:03.431984 25015 ts_tablet_manager.cc:1905] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:03.433946 25013 ts_tablet_manager.cc:1905] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:03.435119 25015 ts_tablet_manager.cc:1918] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:03.435294 25015 log.cc:1196] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/2cb89b94629a462f860179fe5162bb9d
I20260619 22:30:03.435671 25015 ts_tablet_manager.cc:1939] T 2cb89b94629a462f860179fe5162bb9d P c6b58b2cfbd741ff9171d66174d802d7: Deleting consensus metadata
I20260619 22:30:03.437134 24507 catalog_manager.cc:4841] TS c6b58b2cfbd741ff9171d66174d802d7 (127.5.176.194:41609): tablet 2cb89b94629a462f860179fe5162bb9d (table test-table [id=26208077dbfd45e88c48636cdd03a2cb]) successfully deleted
I20260619 22:30:03.437605 25013 ts_tablet_manager.cc:1918] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:03.437733 25013 log.cc:1196] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/2cb89b94629a462f860179fe5162bb9d
I20260619 22:30:03.438205 25013 ts_tablet_manager.cc:1939] T 2cb89b94629a462f860179fe5162bb9d P 83919f23bdc943f7859dfa6c681a7a12: Deleting consensus metadata
W20260619 22:30:03.438679 24508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:46095 (error 108)
I20260619 22:30:03.439839 24509 catalog_manager.cc:4841] TS 83919f23bdc943f7859dfa6c681a7a12 (127.5.176.195:45357): tablet 2cb89b94629a462f860179fe5162bb9d (table test-table [id=26208077dbfd45e88c48636cdd03a2cb]) successfully deleted
I20260619 22:30:03.440049 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24714
I20260619 22:30:03.441555 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:03.441277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:46095 (local address 127.0.0.1:59064)
0619 22:30:03.441413 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:03.441482 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 22:30:03.441758 24508 catalog_manager.cc:4584] TS f56a20f28b044e9a93309303ceffd816 (127.5.176.193:46095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cb89b94629a462f860179fe5162bb9d: Network error: Client connection negotiation failed: client connection to 127.5.176.193:46095: connect: Connection refused (error 111)
I20260619 22:30:03.446537 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24845
I20260619 22:30:03.451704 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 24492
2026-06-19T22:30:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1264 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:03 dist-test-slave-bt27 krb5kdc[25019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:03 dist-test-slave-bt27 krb5kdc[25019](info): set up 2 sockets
Jun 19 22:30:03 dist-test-slave-bt27 krb5kdc[25019](info): commencing operation
krb5kdc: starting...
W20260619 22:30:04.498086 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:04 dist-test-slave-bt27 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:04.625254 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:36433
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:36433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:04.739133 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:04.739468 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:04.739557 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:04.743602 25032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:04.743721 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:04.743772 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:04.743813 25032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:04.743853 25032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:04.749265 25032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:36433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:36433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:04.750465 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:04.751502 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:04.760409 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:04.760555 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:04.760623 25032 server_base.cc:1034] running on GCE node
W20260619 22:30:04.760434 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:04.761510 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:04.762812 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:04.764024 25032 hybrid_clock.cc:648] HybridClock initialized: now 1781908204764004 us; error 31 us; skew 500 ppm
Jun 19 22:30:04 dist-test-slave-bt27 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:04.766965 25032 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:04.767989 25032 webserver.cc:460] Webserver started at http://127.5.176.254:43751/ using document root <none> and password file <none>
I20260619 22:30:04.768641 25032 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:04.768744 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:04.768999 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:04.771370 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "4744e7f97d8e44c4805f6ba21e14e3b0"
format_stamp: "Formatted at 2026-06-19 22:30:04 on dist-test-slave-bt27"
server_key: "8ecab2341c339340e275ed6a29736b48"
server_key_iv: "6c158a9e5069e34169ae2845bf513b7c"
server_key_version: "encryptionkey@0"
I20260619 22:30:04.772013 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "4744e7f97d8e44c4805f6ba21e14e3b0"
format_stamp: "Formatted at 2026-06-19 22:30:04 on dist-test-slave-bt27"
server_key: "8ecab2341c339340e275ed6a29736b48"
server_key_iv: "6c158a9e5069e34169ae2845bf513b7c"
server_key_version: "encryptionkey@0"
I20260619 22:30:04.776145 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:04.778813 25047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:04.780277 25032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:04.780468 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "4744e7f97d8e44c4805f6ba21e14e3b0"
format_stamp: "Formatted at 2026-06-19 22:30:04 on dist-test-slave-bt27"
server_key: "8ecab2341c339340e275ed6a29736b48"
server_key_iv: "6c158a9e5069e34169ae2845bf513b7c"
server_key_version: "encryptionkey@0"
I20260619 22:30:04.780624 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:04.792596 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:04.795794 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:04.795965 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:04.802795 25032 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:36433
I20260619 22:30:04.802893 25099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:36433 every 8 connection(s)
I20260619 22:30:04.803830 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:04.806684 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:04.811723 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25032
I20260619 22:30:04.811892 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:04.812193 5827 external_mini_cluster.cc:1428] Setting key a4e0981e3619b96ac85fc74003594162
I20260619 22:30:04.812405 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0: Bootstrap starting.
I20260619 22:30:04.815016 25100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:04.815954 25100 log.cc:824] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:04.818169 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0: No bootstrap required, opened a new log
Jun 19 22:30:04 dist-test-slave-bt27 krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:04.822918 25100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36433 } }
I20260619 22:30:04.823246 25100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:04.823318 25100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4744e7f97d8e44c4805f6ba21e14e3b0, State: Initialized, Role: FOLLOWER
I20260619 22:30:04.823854 25100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36433 } }
I20260619 22:30:04.824048 25100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:04.824128 25100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:04.824232 25100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:04.825529 25100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36433 } }
I20260619 22:30:04.825927 25100 leader_election.cc:302] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4744e7f97d8e44c4805f6ba21e14e3b0; no voters:
I20260619 22:30:04.826382 25100 leader_election.cc:288] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:04.826460 25105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:04.826651 25105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [term 1 LEADER]: Becoming Leader. State: Replica: 4744e7f97d8e44c4805f6ba21e14e3b0, State: Running, Role: LEADER
I20260619 22:30:04.826999 25105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36433 } }
I20260619 22:30:04.827673 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:04.813526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.5.176.254:36433)
0619 22:30:04.813958 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:04.813973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:04.814008 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:04.814716 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:04.814732 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:04.815059 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:04.815394 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:04.816294 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:04.817253 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:04.818455 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:04.818996 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:04.819142 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:04.822029 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:04.822037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:04.822055 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:04.822090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:04.824186 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:04.824703 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:04.824709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:04.824715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:04.824860 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:04.825197 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:04.825200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:04.825202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:04.825791 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:04.825998 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:04.826364 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 22:30:04.826838 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":155,"threads_started":1}
I20260619 22:30:04.827739 25100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:04.828193 25105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36433 } } }
I20260619 22:30:04.828330 25105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:04.828547 25105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4744e7f97d8e44c4805f6ba21e14e3b0. Latest consensus state: current_term: 1 leader_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744e7f97d8e44c4805f6ba21e14e3b0" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 36433 } } }
I20260619 22:30:04.828651 25105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:04.830195 25109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:04.832859 25109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:04.838847 25109 catalog_manager.cc:1347] Generated new cluster ID: 5a428940c9aa42d790d662850fcebd44
I20260619 22:30:04.838944 25109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:04.863035 25109 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:04.864085 25109 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:04.870827 25109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0: Generated new TSK 0
I20260619 22:30:04.871589 25109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
Entry for principal kudu/127.5.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab.
I20260619 22:30:04.908826 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36433
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:05.022063 25126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:05.022377 25126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:05.022423 25126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:05.026713 25126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:05.026784 25126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:05.026846 25126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:30:05.032769 25126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.193.keytab
--principal=kudu/127.5.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:05.033876 25126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:05.034981 25126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:05.044582 25131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:05.044636 25132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.044780 25126 server_base.cc:1034] running on GCE node
W20260619 22:30:05.044584 25134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.045289 25126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:05.045995 25126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:05.047202 25126 hybrid_clock.cc:648] HybridClock initialized: now 1781908205047182 us; error 31 us; skew 500 ppm
Jun 19 22:30:05 dist-test-slave-bt27 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908205, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:05.050251 25126 init.cc:377] Logged in from keytab as kudu/127.5.176.193@KRBTEST.COM (short username kudu)
I20260619 22:30:05.051199 25126 webserver.cc:460] Webserver started at http://127.5.176.193:37879/ using document root <none> and password file <none>
I20260619 22:30:05.051774 25126 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:05.051834 25126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:05.052052 25126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:05.054095 25126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "9c2e9d9e69764068a4fd58d3f2cd258c"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "47448678021d1108ec4411b9d2bee510"
server_key_iv: "e02efff6d0d6d92e91dde5b4ebdeed34"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.054764 25126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "9c2e9d9e69764068a4fd58d3f2cd258c"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "47448678021d1108ec4411b9d2bee510"
server_key_iv: "e02efff6d0d6d92e91dde5b4ebdeed34"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.058743 25126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:05.061329 25141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.062721 25126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:05.062837 25126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "9c2e9d9e69764068a4fd58d3f2cd258c"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "47448678021d1108ec4411b9d2bee510"
server_key_iv: "e02efff6d0d6d92e91dde5b4ebdeed34"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.062994 25126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:05.079628 25126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:05.082746 25126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:05.082962 25126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:05.083596 25126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:05.084497 25126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:05.084544 25126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.084617 25126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:05.084650 25126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.093459 25126 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:43765
I20260619 22:30:05.093482 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:43765 every 8 connection(s)
I20260619 22:30:05.094378 25126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:30:05.095547 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25126
I20260619 22:30:05.095690 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:30:05.095973 5827 external_mini_cluster.cc:1428] Setting key 6d6eac5228373b22c66e3b93f894cf3a
Jun 19 22:30:05 dist-test-slave-bt27 krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908205, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.193@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.194@KRBTEST.COM" created.
I20260619 22:30:05.110244 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.096331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:49629 (local address 127.5.176.254:36433)
0619 22:30:05.096469 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.096473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.097669 (+ 1196us) server_negotiation.cc:407] Connection header received
0619 22:30:05.098620 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.098623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.098675 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.098752 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:05.099862 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.100404 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.102224 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.102656 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.102740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.106015 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:05.106022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:05.106028 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:05.106064 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:05.107734 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.108311 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.108314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.108315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.108408 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.108898 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.108901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.108902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.109116 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:05.109227 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.109890 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.110042 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 22:30:05.111527 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.096781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:36433 (local address 127.5.176.193:49629)
0619 22:30:05.097533 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.097576 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.098430 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.098915 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.098921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.099260 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:05.099714 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.099726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.100532 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.100538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.102041 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.102056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.102974 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.102980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.103217 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.103967 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:05.103995 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:05.105801 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:05.107901 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:05.107907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:05.107923 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:05.108207 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:05.108530 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:05.108534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:05.108535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:05.108790 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:05.109241 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:05.109247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:05.109625 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.109859 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.110305 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":112,"threads_started":1}
I20260619 22:30:05.112936 25255 heartbeater.cc:347] Connected to a master server at 127.5.176.254:36433
I20260619 22:30:05.113199 25255 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:05.113729 25255 heartbeater.cc:511] Master 127.5.176.254:36433 requested a full tablet report, sending...
I20260619 22:30:05.115315 25064 ts_manager.cc:194] Registered new tserver with Master: 9c2e9d9e69764068a4fd58d3f2cd258c (127.5.176.193:43765)
I20260619 22:30:05.116480 25064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.193@KRBTEST.COM'} at 127.5.176.193:49629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
Entry for principal kudu/127.5.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab.
I20260619 22:30:05.125875 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36433
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:05.241479 25260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:05.241828 25260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:05.241890 25260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:05.246024 25260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:05.246080 25260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:05.246158 25260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:30:05.251636 25260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.194.keytab
--principal=kudu/127.5.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:05.252743 25260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:05.253700 25260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:05.264142 25268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:05.264158 25266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.264242 25260 server_base.cc:1034] running on GCE node
W20260619 22:30:05.264158 25265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.264943 25260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:05.265615 25260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:05.266801 25260 hybrid_clock.cc:648] HybridClock initialized: now 1781908205266743 us; error 47 us; skew 500 ppm
Jun 19 22:30:05 dist-test-slave-bt27 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908205, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:05.269706 25260 init.cc:377] Logged in from keytab as kudu/127.5.176.194@KRBTEST.COM (short username kudu)
I20260619 22:30:05.270730 25260 webserver.cc:460] Webserver started at http://127.5.176.194:35165/ using document root <none> and password file <none>
I20260619 22:30:05.271349 25260 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:05.271446 25260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:05.271684 25260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:05.273873 25260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "ac96a4f3cc66428485b2da77fb495dc2"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "beffd9da941b4bb870835bf1f6ae3adc"
server_key_iv: "474d46db45fa28127558ad04ed9a691a"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.274557 25260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "ac96a4f3cc66428485b2da77fb495dc2"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "beffd9da941b4bb870835bf1f6ae3adc"
server_key_iv: "474d46db45fa28127558ad04ed9a691a"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.278497 25260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:30:05.281078 25275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.282368 25260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:05.282500 25260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "ac96a4f3cc66428485b2da77fb495dc2"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "beffd9da941b4bb870835bf1f6ae3adc"
server_key_iv: "474d46db45fa28127558ad04ed9a691a"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.282622 25260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:05.306106 25260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:05.309672 25260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:05.309891 25260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:05.310587 25260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:05.311481 25260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:05.311554 25260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.311626 25260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:05.311676 25260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.320670 25260 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:42393
I20260619 22:30:05.320700 25388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:42393 every 8 connection(s)
I20260619 22:30:05.321704 25260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:30:05.323273 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25260
I20260619 22:30:05.323424 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:30:05.323799 5827 external_mini_cluster.cc:1428] Setting key 94d5f3f0be3161925aa971dbdc8410f6
Jun 19 22:30:05 dist-test-slave-bt27 krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908205, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.194@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.176.195@KRBTEST.COM; defaulting to no policy
I20260619 22:30:05.337713 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.323842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:44047 (local address 127.5.176.254:36433)
0619 22:30:05.323990 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.323995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.324991 (+ 996us) server_negotiation.cc:407] Connection header received
0619 22:30:05.326217 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.326220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.326284 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.326354 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:05.327997 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.328647 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.330240 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.330598 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.330719 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.333778 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:05.333782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:05.333784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:05.333811 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:05.335376 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.335917 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.335920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.335921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.336083 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.336480 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.336482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.336484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.336682 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:05.336764 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.337360 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.337515 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.195@KRBTEST.COM" created.
I20260619 22:30:05.338692 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.324284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:36433 (local address 127.5.176.194:44047)
0619 22:30:05.324831 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.324878 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.325928 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.326561 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.326573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.327123 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:05.327842 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.327857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.328789 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.328792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.330065 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.330075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.330767 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.330771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.330991 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.331733 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:05.331751 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:05.333555 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:05.335500 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:05.335506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:05.335520 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:05.335788 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:05.336214 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:05.336217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:05.336218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:05.336383 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:05.336810 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:05.336816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:05.337107 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.337295 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.337580 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":186,"threads_started":1}
I20260619 22:30:05.339941 25389 heartbeater.cc:347] Connected to a master server at 127.5.176.254:36433
I20260619 22:30:05.340253 25389 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:05.340885 25389 heartbeater.cc:511] Master 127.5.176.254:36433 requested a full tablet report, sending...
I20260619 22:30:05.341964 25064 ts_manager.cc:194] Registered new tserver with Master: ac96a4f3cc66428485b2da77fb495dc2 (127.5.176.194:42393)
I20260619 22:30:05.342636 25064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.194@KRBTEST.COM'} at 127.5.176.194:44047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
Entry for principal kudu/127.5.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab.
I20260619 22:30:05.355348 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36433
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:05.470623 25394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:05.470973 25394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:05.471045 25394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:05.475302 25394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:05.475378 25394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:05.475440 25394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:30:05.480924 25394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.195.keytab
--principal=kudu/127.5.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:36433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:05.482120 25394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:05.483145 25394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:05.493547 25399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:05.493530 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:05.493530 25402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.493670 25394 server_base.cc:1034] running on GCE node
I20260619 22:30:05.494294 25394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:05.495003 25394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:05.496187 25394 hybrid_clock.cc:648] HybridClock initialized: now 1781908205496168 us; error 30 us; skew 500 ppm
Jun 19 22:30:05 dist-test-slave-bt27 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908205, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:05.499188 25394 init.cc:377] Logged in from keytab as kudu/127.5.176.195@KRBTEST.COM (short username kudu)
I20260619 22:30:05.500134 25394 webserver.cc:460] Webserver started at http://127.5.176.195:42893/ using document root <none> and password file <none>
I20260619 22:30:05.500715 25394 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:05.500840 25394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:05.501078 25394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:05.503298 25394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "04caa7d6778a4cf4be5a588b7e8809ae"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "ec7417a3fc5536c4b49e66c3fb397b38"
server_key_iv: "f099257bda03405fd51c5b5b5a5d52d1"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.503931 25394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "04caa7d6778a4cf4be5a588b7e8809ae"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "ec7417a3fc5536c4b49e66c3fb397b38"
server_key_iv: "f099257bda03405fd51c5b5b5a5d52d1"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.508173 25394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:05.510926 25409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.512300 25394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:05.512458 25394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "04caa7d6778a4cf4be5a588b7e8809ae"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "ec7417a3fc5536c4b49e66c3fb397b38"
server_key_iv: "f099257bda03405fd51c5b5b5a5d52d1"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.512589 25394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:05.537307 25394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:05.540709 25394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:05.540943 25394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:05.541565 25394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:05.542517 25394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:05.542587 25394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.542657 25394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:05.542706 25394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.552145 25394 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:37361
I20260619 22:30:05.552208 25522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:37361 every 8 connection(s)
I20260619 22:30:05.553112 25394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
Jun 19 22:30:05 dist-test-slave-bt27 krb5kdc[25019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908205, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.195@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:05.562990 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25394
I20260619 22:30:05.563145 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:30:05.563478 5827 external_mini_cluster.cc:1428] Setting key c65e3d89d67f1cee9eb44ce9d1135112
I20260619 22:30:05.567166 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.555052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:53337 (local address 127.5.176.254:36433)
0619 22:30:05.555196 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.555200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.556037 (+ 837us) server_negotiation.cc:407] Connection header received
0619 22:30:05.556940 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.556944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.557002 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.557083 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:05.558279 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.558827 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.560004 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.560237 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.560354 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.563220 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:05.563227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:05.563232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:05.563269 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:05.565041 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.565566 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.565571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.565574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.565669 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.566054 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.566056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.566058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.566317 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:05.566398 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.566879 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.566988 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:30:05.568074 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.555428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:36433 (local address 127.5.176.195:53337)
0619 22:30:05.555872 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.555913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.556727 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.557249 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.557255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.557620 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:05.558100 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.558113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.558966 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.558983 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.559860 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.559869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.560399 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.560403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.560586 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.561234 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:05.561251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:05.562984 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:05.565148 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:05.565153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:05.565166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:05.565432 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:05.565771 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:05.565775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:05.565777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:05.565929 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:05.566408 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:05.566413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:05.566664 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.566832 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.567156 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260619 22:30:05.569195 25523 heartbeater.cc:347] Connected to a master server at 127.5.176.254:36433
I20260619 22:30:05.569432 25523 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:05.569922 25523 heartbeater.cc:511] Master 127.5.176.254:36433 requested a full tablet report, sending...
I20260619 22:30:05.570986 25064 ts_manager.cc:194] Registered new tserver with Master: 04caa7d6778a4cf4be5a588b7e8809ae (127.5.176.195:37361)
I20260619 22:30:05.571574 25064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.176.195@KRBTEST.COM'} at 127.5.176.195:53337
I20260619 22:30:05.577422 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:30:05.587168 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.579427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.5.176.254:36433)
0619 22:30:05.579561 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.579564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.579721 (+ 157us) server_negotiation.cc:407] Connection header received
0619 22:30:05.579818 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.579820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.579868 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.579934 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:05.580617 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.581183 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.581986 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.582317 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.582400 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.583770 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:05.583774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:05.583776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:05.583802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:05.585347 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.585779 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.585786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.585791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.585927 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:05.586251 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:05.586254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:05.586255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:05.586503 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:05.586613 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.586810 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.586968 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:30:05.590549 25064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:30:05.593804 25064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:30:05.609458 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.602818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:42393 (local address 127.0.0.1:55714)
0619 22:30:05.603349 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.603407 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.603549 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.604432 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.604438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.604470 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:05.604652 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.604660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.606039 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.606046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.607547 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.607557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.608862 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.608866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.608998 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.609048 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.609151 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.609253 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":121,"thread_start_us":71,"threads_started":1}
I20260619 22:30:05.609510 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.602928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:37361 (local address 127.0.0.1:46924)
0619 22:30:05.604327 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.604339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.604452 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.604948 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.604952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.604969 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:05.605063 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.605070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.606319 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.606323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.607684 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.607695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.609072 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.609076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.609190 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.609218 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.609302 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.609374 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"mutex_wait_us":96,"thread_start_us":114,"threads_started":1}
I20260619 22:30:05.609606 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.603627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.5.176.195:37361)
0619 22:30:05.604029 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.604042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.604564 (+ 522us) server_negotiation.cc:407] Connection header received
0619 22:30:05.604642 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.604646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.604823 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.604978 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:05.605159 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.606162 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.607861 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.608706 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.608840 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.609009 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.609353 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.609454 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":116,"threads_started":1}
I20260619 22:30:05.609695 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.602975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.5.176.194:42393)
0619 22:30:05.603875 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.603890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.603919 (+ 29us) server_negotiation.cc:407] Connection header received
0619 22:30:05.604008 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.604014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.604236 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.604431 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:05.604788 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.605904 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.607687 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.608695 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.608960 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.609099 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.609203 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.609338 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":177,"threads_started":1}
I20260619 22:30:05.611001 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.603945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.5.176.193:43765)
0619 22:30:05.604322 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.604339 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.604753 (+ 414us) server_negotiation.cc:407] Connection header received
0619 22:30:05.604848 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.604853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.605040 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.605201 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:05.605598 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.606748 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.608145 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.608699 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.608831 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.609010 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.610695 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.610817 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260619 22:30:05.613104 25323 tablet_service.cc:1467] Processing CreateTablet for tablet 7074151c08324b8da40867ecc84a7f3a (DEFAULT_TABLE table=test-table [id=fb2c8940b867448193e1a7da37ee639f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:05.613106 25457 tablet_service.cc:1467] Processing CreateTablet for tablet 7074151c08324b8da40867ecc84a7f3a (DEFAULT_TABLE table=test-table [id=fb2c8940b867448193e1a7da37ee639f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:05.613976 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.603544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:43765 (local address 127.0.0.1:41166)
0619 22:30:05.604514 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.604529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.604627 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.605360 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.605363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.605381 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:05.605464 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.605470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.606894 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.606898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.608029 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.608037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.608832 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.608836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.610524 (+ 1688us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.610550 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.610632 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.610706 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":61,"thread_start_us":97,"threads_started":1}
I20260619 22:30:05.614353 25457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7074151c08324b8da40867ecc84a7f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:05.615168 25323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7074151c08324b8da40867ecc84a7f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:05.617029 25189 tablet_service.cc:1467] Processing CreateTablet for tablet 7074151c08324b8da40867ecc84a7f3a (DEFAULT_TABLE table=test-table [id=fb2c8940b867448193e1a7da37ee639f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:05.618055 25189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7074151c08324b8da40867ecc84a7f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:05.622195 25540 tablet_bootstrap.cc:492] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Bootstrap starting.
I20260619 22:30:05.622627 25541 tablet_bootstrap.cc:492] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Bootstrap starting.
I20260619 22:30:05.624495 25542 tablet_bootstrap.cc:492] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: Bootstrap starting.
I20260619 22:30:05.626632 25542 tablet_bootstrap.cc:654] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:05.626828 25541 tablet_bootstrap.cc:654] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:05.627494 25542 log.cc:824] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:05.627560 25540 tablet_bootstrap.cc:654] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:05.627697 25541 log.cc:824] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:05.628386 25540 log.cc:824] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:05.629706 25542 tablet_bootstrap.cc:492] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: No bootstrap required, opened a new log
I20260619 22:30:05.629824 25541 tablet_bootstrap.cc:492] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: No bootstrap required, opened a new log
I20260619 22:30:05.629956 25542 ts_tablet_manager.cc:1397] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 22:30:05.630020 25541 ts_tablet_manager.cc:1397] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 22:30:05.630319 25540 tablet_bootstrap.cc:492] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: No bootstrap required, opened a new log
I20260619 22:30:05.630486 25540 ts_tablet_manager.cc:1397] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 22:30:05.633919 25542 raft_consensus.cc:348] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.633960 25541 raft_consensus.cc:348] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.634208 25542 raft_consensus.cc:374] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:05.634042 25540 raft_consensus.cc:348] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.634270 25542 raft_consensus.cc:729] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2e9d9e69764068a4fd58d3f2cd258c, State: Initialized, Role: FOLLOWER
I20260619 22:30:05.634284 25540 raft_consensus.cc:374] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:05.634274 25541 raft_consensus.cc:374] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:05.634330 25541 raft_consensus.cc:729] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac96a4f3cc66428485b2da77fb495dc2, State: Initialized, Role: FOLLOWER
I20260619 22:30:05.634336 25540 raft_consensus.cc:729] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04caa7d6778a4cf4be5a588b7e8809ae, State: Initialized, Role: FOLLOWER
I20260619 22:30:05.634747 25542 consensus_queue.cc:260] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.634792 25540 consensus_queue.cc:260] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.634795 25541 consensus_queue.cc:260] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.635468 25255 heartbeater.cc:503] Master 127.5.176.254:36433 was elected leader, sending a full tablet report...
I20260619 22:30:05.635807 25540 ts_tablet_manager.cc:1428] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:30:05.635830 25542 ts_tablet_manager.cc:1428] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 22:30:05.635928 25523 heartbeater.cc:503] Master 127.5.176.254:36433 was elected leader, sending a full tablet report...
I20260619 22:30:05.636031 25389 heartbeater.cc:503] Master 127.5.176.254:36433 was elected leader, sending a full tablet report...
I20260619 22:30:05.638398 25541 ts_tablet_manager.cc:1428] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 22:30:05.661072 25548 raft_consensus.cc:482] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:30:05.661312 25548 raft_consensus.cc:504] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.662652 25548 leader_election.cc:288] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04caa7d6778a4cf4be5a588b7e8809ae (127.5.176.195:37361), 9c2e9d9e69764068a4fd58d3f2cd258c (127.5.176.193:43765)
I20260619 22:30:05.667702 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.663050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:58497 (local address 127.5.176.193:43765)
0619 22:30:05.663203 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.663209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.663257 (+ 48us) server_negotiation.cc:407] Connection header received
0619 22:30:05.663315 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.663317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.663366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.663446 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:05.663797 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.664642 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.666282 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.666929 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.667071 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.667129 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.667486 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.667566 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 22:30:05.667702 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.663019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:52615 (local address 127.5.176.195:37361)
0619 22:30:05.663210 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.663214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.663527 (+ 313us) server_negotiation.cc:407] Connection header received
0619 22:30:05.663644 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.663647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.663688 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.663832 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:05.664059 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.664780 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.666282 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.666948 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.667071 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.667129 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.667505 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.667581 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 22:30:05.667719 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.662914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:43765 (local address 127.5.176.194:58497)
0619 22:30:05.663114 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.663132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.663229 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.663494 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.663497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.663536 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:05.663676 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.663682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.664769 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.664772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.666176 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.666181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.667089 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.667093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.667297 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.667362 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.667453 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.667538 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":84,"spinlock_wait_cycles":27264}
I20260619 22:30:05.667719 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.662932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:37361 (local address 127.5.176.194:52615)
0619 22:30:05.663397 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.663407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:05.663542 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:05.663853 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:05.663855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:05.663870 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:05.663948 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.663951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.664910 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.664913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.666169 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:05.666181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.667089 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.667093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:05.667293 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.667406 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 22:30:05.667476 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 22:30:05.667538 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":90,"spinlock_wait_cycles":6528,"thread_start_us":144,"threads_started":1}
I20260619 22:30:05.668653 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7074151c08324b8da40867ecc84a7f3a" candidate_uuid: "ac96a4f3cc66428485b2da77fb495dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" is_pre_election: true
I20260619 22:30:05.668717 25209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7074151c08324b8da40867ecc84a7f3a" candidate_uuid: "ac96a4f3cc66428485b2da77fb495dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" is_pre_election: true
I20260619 22:30:05.669035 25477 raft_consensus.cc:2454] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac96a4f3cc66428485b2da77fb495dc2 in term 0.
I20260619 22:30:05.669055 25209 raft_consensus.cc:2454] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac96a4f3cc66428485b2da77fb495dc2 in term 0.
I20260619 22:30:05.669584 25279 leader_election.cc:302] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04caa7d6778a4cf4be5a588b7e8809ae, ac96a4f3cc66428485b2da77fb495dc2; no voters:
I20260619 22:30:05.669940 25548 raft_consensus.cc:2781] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:30:05.670068 25548 raft_consensus.cc:482] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:30:05.670156 25548 raft_consensus.cc:3037] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:05.671252 25548 raft_consensus.cc:504] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.671566 25548 leader_election.cc:288] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 04caa7d6778a4cf4be5a588b7e8809ae (127.5.176.195:37361), 9c2e9d9e69764068a4fd58d3f2cd258c (127.5.176.193:43765)
I20260619 22:30:05.671953 25209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7074151c08324b8da40867ecc84a7f3a" candidate_uuid: "ac96a4f3cc66428485b2da77fb495dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c"
I20260619 22:30:05.672081 25209 raft_consensus.cc:3037] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:05.672154 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7074151c08324b8da40867ecc84a7f3a" candidate_uuid: "ac96a4f3cc66428485b2da77fb495dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04caa7d6778a4cf4be5a588b7e8809ae"
I20260619 22:30:05.672330 25477 raft_consensus.cc:3037] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:05.673019 25209 raft_consensus.cc:2454] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac96a4f3cc66428485b2da77fb495dc2 in term 1.
I20260619 22:30:05.673346 25276 leader_election.cc:302] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c2e9d9e69764068a4fd58d3f2cd258c, ac96a4f3cc66428485b2da77fb495dc2; no voters:
I20260619 22:30:05.673583 25548 raft_consensus.cc:2781] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:05.673754 25477 raft_consensus.cc:2454] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac96a4f3cc66428485b2da77fb495dc2 in term 1.
I20260619 22:30:05.673879 25548 raft_consensus.cc:686] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 1 LEADER]: Becoming Leader. State: Replica: ac96a4f3cc66428485b2da77fb495dc2, State: Running, Role: LEADER
I20260619 22:30:05.674310 25548 consensus_queue.cc:237] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } }
I20260619 22:30:05.677507 25064 catalog_manager.cc:5495] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac96a4f3cc66428485b2da77fb495dc2 (127.5.176.194). New cstate: current_term: 1 leader_uuid: "ac96a4f3cc66428485b2da77fb495dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac96a4f3cc66428485b2da77fb495dc2" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 42393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20260619 22:30:05.689425 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.685813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.5.176.194:42393)
0619 22:30:05.685982 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.685986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.685999 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:30:05.686071 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.686073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.686126 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.686279 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:05.686524 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.687112 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.687923 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.688132 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.688214 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.688502 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:05.688941 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:05.689037 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.689201 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.689281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 22:30:05.695530 25209 raft_consensus.cc:1261] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 1 FOLLOWER]: Refusing update from remote peer ac96a4f3cc66428485b2da77fb495dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:05.695530 25477 raft_consensus.cc:1261] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 1 FOLLOWER]: Refusing update from remote peer ac96a4f3cc66428485b2da77fb495dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:05.696328 25550 consensus_queue.cc:1035] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c2e9d9e69764068a4fd58d3f2cd258c" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:05.696590 25548 consensus_queue.cc:1035] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04caa7d6778a4cf4be5a588b7e8809ae" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:05.705879 25555 mvcc.cc:205] Tried to move back new op lower bound from 7298696010520711168 to 7298696010444234752. Current Snapshot: MvccSnapshot[applied={T|T < 7298696010520711168}]
I20260619 22:30:05.706226 25554 mvcc.cc:205] Tried to move back new op lower bound from 7298696010520711168 to 7298696010444234752. Current Snapshot: MvccSnapshot[applied={T|T < 7298696010520711168}]
I20260619 22:30:05.713441 25062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:05.713641 25062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:05.716185 25062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4744e7f97d8e44c4805f6ba21e14e3b0: Sending DeleteTablet for 3 replicas of tablet 7074151c08324b8da40867ecc84a7f3a
I20260619 22:30:05.717084 25457 tablet_service.cc:1514] Processing DeleteTablet for tablet 7074151c08324b8da40867ecc84a7f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:05 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:46924
I20260619 22:30:05.717139 25189 tablet_service.cc:1514] Processing DeleteTablet for tablet 7074151c08324b8da40867ecc84a7f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:05 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:41166
I20260619 22:30:05.717669 25565 tablet_replica.cc:330] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: stopping tablet replica
I20260619 22:30:05.717693 25323 tablet_service.cc:1514] Processing DeleteTablet for tablet 7074151c08324b8da40867ecc84a7f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:05 UTC) from {username='kudu', principal='kudu/127.5.176.254@KRBTEST.COM'} at 127.0.0.1:55714
I20260619 22:30:05.718057 25565 raft_consensus.cc:2229] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:05.718245 25566 tablet_replica.cc:330] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: stopping tablet replica
I20260619 22:30:05.718436 25565 raft_consensus.cc:2258] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:05.718626 25566 raft_consensus.cc:2229] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:30:05.718751 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25126
I20260619 22:30:05.718827 25564 tablet_replica.cc:330] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c: stopping tablet replica
I20260619 22:30:05.719136 25564 raft_consensus.cc:2229] T 7074151c08324b8da40867ecc84a7f3a P 9c2e9d9e69764068a4fd58d3f2cd258c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:05.719148 25566 raft_consensus.cc:2258] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:05.721194 25566 ts_tablet_manager.cc:1905] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:05.723201 25565 ts_tablet_manager.cc:1905] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:05.724339 25566 ts_tablet_manager.cc:1918] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:05.724472 25566 log.cc:1196] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/7074151c08324b8da40867ecc84a7f3a
I20260619 22:30:05.724799 25566 ts_tablet_manager.cc:1939] T 7074151c08324b8da40867ecc84a7f3a P ac96a4f3cc66428485b2da77fb495dc2: Deleting consensus metadata
I20260619 22:30:05.725868 25565 ts_tablet_manager.cc:1918] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:05.725971 25565 log.cc:1196] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/7074151c08324b8da40867ecc84a7f3a
I20260619 22:30:05.726012 25048 catalog_manager.cc:4841] TS ac96a4f3cc66428485b2da77fb495dc2 (127.5.176.194:42393): tablet 7074151c08324b8da40867ecc84a7f3a (table test-table [id=fb2c8940b867448193e1a7da37ee639f]) successfully deleted
I20260619 22:30:05.726346 25565 ts_tablet_manager.cc:1939] T 7074151c08324b8da40867ecc84a7f3a P 04caa7d6778a4cf4be5a588b7e8809ae: Deleting consensus metadata
I20260619 22:30:05.727602 25051 catalog_manager.cc:4841] TS 04caa7d6778a4cf4be5a588b7e8809ae (127.5.176.195:37361): tablet 7074151c08324b8da40867ecc84a7f3a (table test-table [id=fb2c8940b867448193e1a7da37ee639f]) successfully deleted
W20260619 22:30:05.732913 25048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:43765 (error 108)
I20260619 22:30:05.733153 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25260
I20260619 22:30:05.735358 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.735057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:43765 (local address 127.0.0.1:41172)
0619 22:30:05.735204 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:05.735274 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 22:30:05.735541 25048 catalog_manager.cc:4584] TS 9c2e9d9e69764068a4fd58d3f2cd258c (127.5.176.193:43765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7074151c08324b8da40867ecc84a7f3a: Network error: Client connection negotiation failed: client connection to 127.5.176.193:43765: connect: Connection refused (error 111)
I20260619 22:30:05.739449 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25394
I20260619 22:30:05.745390 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25032
2026-06-19T22:30:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2294 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T22:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:05Z Disabled control of system clock
I20260619 22:30:05.776427 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43093
--webserver_interface=127.5.176.254
--webserver_port=0
--builtin_ntp_servers=127.5.176.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:43093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:30:05.887456 25571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:05.887758 25571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:05.887797 25571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:05.892024 25571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:05.892092 25571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:05.892108 25571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:05.892125 25571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:05.892141 25571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:05.897269 25571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:43093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:05.898458 25571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:05.899906 25571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:05.908679 25579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:05.908694 25576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.908762 25571 server_base.cc:1034] running on GCE node
W20260619 22:30:05.908680 25577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:05.909703 25571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:05.911023 25571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:05.912247 25571 hybrid_clock.cc:648] HybridClock initialized: now 1781908205912219 us; error 43 us; skew 500 ppm
I20260619 22:30:05.914361 25571 webserver.cc:460] Webserver started at http://127.5.176.254:41087/ using document root <none> and password file <none>
I20260619 22:30:05.914976 25571 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:05.915074 25571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:05.915314 25571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:05.917387 25571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "d8ac5b085cd94778b65e2a625d01386f"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "072958f0971f6c96a8d394630cccbc9f"
server_key_iv: "2fa1af096889f389c74d7110d9139fd9"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.917976 25571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "d8ac5b085cd94778b65e2a625d01386f"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "072958f0971f6c96a8d394630cccbc9f"
server_key_iv: "2fa1af096889f389c74d7110d9139fd9"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.921862 25571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 22:30:05.924386 25585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:05.925625 25571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:30:05.925772 25571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "d8ac5b085cd94778b65e2a625d01386f"
format_stamp: "Formatted at 2026-06-19 22:30:05 on dist-test-slave-bt27"
server_key: "072958f0971f6c96a8d394630cccbc9f"
server_key_iv: "2fa1af096889f389c74d7110d9139fd9"
server_key_version: "encryptionkey@0"
I20260619 22:30:05.925895 25571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:05.937554 25571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:05.938287 25571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:05.938479 25571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:05.945677 25571 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:43093
I20260619 22:30:05.945693 25637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:43093 every 8 connection(s)
I20260619 22:30:05.946699 25571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:05.949893 25638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:05.952370 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25571
I20260619 22:30:05.952513 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:05.952809 5827 external_mini_cluster.cc:1428] Setting key 2d0372dabd3546bc82f9be4926e696b5
I20260619 22:30:05.956413 25638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f: Bootstrap starting.
I20260619 22:30:05.958714 25638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:05.959475 25638 log.cc:824] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:05.961645 25638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f: No bootstrap required, opened a new log
I20260619 22:30:05.962311 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:05.954321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.5.176.254:43093)
0619 22:30:05.954982 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:05.954997 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:05.955032 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:05.955667 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:05.955683 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:05.956083 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:05.956418 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:05.956834 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.957785 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.958752 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:05.959286 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:05.959461 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:05.960380 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:05.960385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:05.960397 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:05.960773 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:05.960828 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:05.960847 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:05.960932 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:05.961117 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 22:30:05.961415 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260619 22:30:05.965690 25638 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ac5b085cd94778b65e2a625d01386f" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43093 } }
I20260619 22:30:05.965945 25638 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:05.966022 25638 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ac5b085cd94778b65e2a625d01386f, State: Initialized, Role: FOLLOWER
I20260619 22:30:05.966537 25638 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ac5b085cd94778b65e2a625d01386f" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43093 } }
I20260619 22:30:05.966704 25638 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:05.966771 25638 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:05.966871 25638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:05.967967 25638 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ac5b085cd94778b65e2a625d01386f" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43093 } }
I20260619 22:30:05.968330 25638 leader_election.cc:302] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ac5b085cd94778b65e2a625d01386f; no voters:
I20260619 22:30:05.968700 25638 leader_election.cc:288] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:05.968871 25643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:05.969147 25643 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [term 1 LEADER]: Becoming Leader. State: Replica: d8ac5b085cd94778b65e2a625d01386f, State: Running, Role: LEADER
I20260619 22:30:05.969485 25643 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ac5b085cd94778b65e2a625d01386f" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43093 } }
I20260619 22:30:05.970057 25638 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:05.971303 25644 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ac5b085cd94778b65e2a625d01386f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ac5b085cd94778b65e2a625d01386f" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43093 } } }
I20260619 22:30:05.971459 25644 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:05.971303 25645 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ac5b085cd94778b65e2a625d01386f. Latest consensus state: current_term: 1 leader_uuid: "d8ac5b085cd94778b65e2a625d01386f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ac5b085cd94778b65e2a625d01386f" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 43093 } } }
I20260619 22:30:05.971580 25645 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:05.972383 25652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:05.975689 25652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:05.982026 25652 catalog_manager.cc:1347] Generated new cluster ID: 1e8af161c18c4da39fea3d18928b5edd
I20260619 22:30:05.982125 25652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:05.998667 25652 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:06.000038 25652 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:06.005484 25652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f: Generated new TSK 0
I20260619 22:30:06.006311 25652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:06.010552 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.193:0
--local_ip_for_outbound_sockets=127.5.176.193
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43093
--builtin_ntp_servers=127.5.176.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 22:30:06.124251 25662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:06.124571 25662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:06.124614 25662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:06.129011 25662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:06.129077 25662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:06.129137 25662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.193
I20260619 22:30:06.134511 25662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.176.193
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.193
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:06.136065 25662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:06.137787 25662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:06.147596 25668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:06.147560 25667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:06.147783 25662 server_base.cc:1034] running on GCE node
W20260619 22:30:06.147560 25670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:06.148293 25662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:06.148957 25662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:06.150211 25662 hybrid_clock.cc:648] HybridClock initialized: now 1781908206150173 us; error 48 us; skew 500 ppm
I20260619 22:30:06.152133 25662 webserver.cc:460] Webserver started at http://127.5.176.193:45621/ using document root <none> and password file <none>
I20260619 22:30:06.152783 25662 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:06.152880 25662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:06.153120 25662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:06.155329 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data/instance:
uuid: "b39439df93584f8dbef694e99f91546f"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "c7a72817767a684b2c8bea4a2681e9bf"
server_key_iv: "f2e83b8ebb5129ac05ce985946a9d3fc"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.155915 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance:
uuid: "b39439df93584f8dbef694e99f91546f"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "c7a72817767a684b2c8bea4a2681e9bf"
server_key_iv: "f2e83b8ebb5129ac05ce985946a9d3fc"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.159902 25662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:06.162612 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.164098 25662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:06.164281 25662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal
uuid: "b39439df93584f8dbef694e99f91546f"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "c7a72817767a684b2c8bea4a2681e9bf"
server_key_iv: "f2e83b8ebb5129ac05ce985946a9d3fc"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.164431 25662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:06.179127 25662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:06.180061 25662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:06.180301 25662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:06.180959 25662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:06.181993 25662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:06.182065 25662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.182175 25662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:06.182224 25662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.192152 25662 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.193:40171
I20260619 22:30:06.192190 25789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.193:40171 every 8 connection(s)
I20260619 22:30:06.193138 25662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/data/info.pb
I20260619 22:30:06.197398 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25662
I20260619 22:30:06.197542 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-0/wal/instance
I20260619 22:30:06.197832 5827 external_mini_cluster.cc:1428] Setting key ed8d023d5c50426106a1c0600cabc395
I20260619 22:30:06.199898 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.194:0
--local_ip_for_outbound_sockets=127.5.176.194
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43093
--builtin_ntp_servers=127.5.176.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:30:06.204124 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.195147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.193:47903 (local address 127.5.176.254:43093)
0619 22:30:06.195297 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.195301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.196385 (+ 1084us) server_negotiation.cc:407] Connection header received
0619 22:30:06.198035 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.198039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.198101 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.198216 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:06.199335 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.199889 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.201698 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.202167 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.202309 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.203481 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:06.203488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:06.203495 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:06.203517 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:06.203546 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:06.203561 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:06.203656 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.203817 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.203903 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:30:06.204978 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.195569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43093 (local address 127.5.176.193:47903)
0619 22:30:06.196211 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.196251 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.197783 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.198418 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.198426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.198569 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:06.199175 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.199194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.200040 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.200044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.201540 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.201550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.202335 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.202338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.202494 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.203186 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:30:06.203188 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:06.203232 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:30:06.203304 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:06.203659 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:06.203666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:06.203671 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.203772 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.204090 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260619 22:30:06.206401 25790 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43093
I20260619 22:30:06.206668 25790 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:06.207204 25790 heartbeater.cc:511] Master 127.5.176.254:43093 requested a full tablet report, sending...
I20260619 22:30:06.208732 25601 ts_manager.cc:194] Registered new tserver with Master: b39439df93584f8dbef694e99f91546f (127.5.176.193:40171)
I20260619 22:30:06.209877 25601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.193:47903
W20260619 22:30:06.314941 25793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:06.315203 25793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:06.315276 25793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:06.319331 25793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:06.319383 25793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:06.319470 25793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.194
I20260619 22:30:06.324241 25793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.176.194
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.194
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:06.325317 25793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:06.326694 25793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:06.336253 25801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:06.336253 25798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:06.336469 25793 server_base.cc:1034] running on GCE node
W20260619 22:30:06.336199 25799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:06.336937 25793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:06.337572 25793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:06.338749 25793 hybrid_clock.cc:648] HybridClock initialized: now 1781908206338735 us; error 31 us; skew 500 ppm
I20260619 22:30:06.343624 25793 webserver.cc:460] Webserver started at http://127.5.176.194:40461/ using document root <none> and password file <none>
I20260619 22:30:06.344337 25793 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:06.344405 25793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:06.344604 25793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:06.346673 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data/instance:
uuid: "e3d8047ba8dc4a909b67595b654d8e39"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "6a80b8d0bfeb8d760320cb79795763c9"
server_key_iv: "56d49bf673ccef84b80b6f6a23bc080c"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.347179 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance:
uuid: "e3d8047ba8dc4a909b67595b654d8e39"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "6a80b8d0bfeb8d760320cb79795763c9"
server_key_iv: "56d49bf673ccef84b80b6f6a23bc080c"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.350730 25793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:06.353399 25807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.354833 25793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:06.354945 25793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal
uuid: "e3d8047ba8dc4a909b67595b654d8e39"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "6a80b8d0bfeb8d760320cb79795763c9"
server_key_iv: "56d49bf673ccef84b80b6f6a23bc080c"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.355048 25793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:06.368673 25793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:06.369381 25793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:06.369537 25793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:06.370261 25793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:06.371292 25793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:06.371366 25793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.371443 25793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:06.371481 25793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.380970 25793 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.194:41085
I20260619 22:30:06.380988 25920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.194:41085 every 8 connection(s)
I20260619 22:30:06.381888 25793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/data/info.pb
I20260619 22:30:06.387199 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25793
I20260619 22:30:06.387334 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal/instance
I20260619 22:30:06.387630 5827 external_mini_cluster.cc:1428] Setting key 40aa92fa95c1a75c290ae153537d49e3
I20260619 22:30:06.389714 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.176.195:0
--local_ip_for_outbound_sockets=127.5.176.195
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43093
--builtin_ntp_servers=127.5.176.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 22:30:06.391556 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.383712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.194:37143 (local address 127.5.176.254:43093)
0619 22:30:06.383867 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.383870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.384871 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 22:30:06.386166 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.386168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.386240 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.386316 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:06.387098 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.387895 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.389308 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.389661 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.389781 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.391053 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:06.391056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:06.391059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:06.391080 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:06.391099 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:06.391107 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:06.391188 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.391356 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.391418 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:30:06.392401 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.384091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43093 (local address 127.5.176.194:37143)
0619 22:30:06.384703 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.384741 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.385978 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.386434 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.386440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.386514 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:06.386968 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.386979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.388083 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.388088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.389193 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.389203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.389873 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.389880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.390100 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.390788 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:30:06.390790 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:06.390830 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:30:06.390893 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:06.391190 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:06.391202 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:06.391211 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.391354 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.391637 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260619 22:30:06.393638 25921 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43093
I20260619 22:30:06.393915 25921 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:06.394506 25921 heartbeater.cc:511] Master 127.5.176.254:43093 requested a full tablet report, sending...
I20260619 22:30:06.395627 25601 ts_manager.cc:194] Registered new tserver with Master: e3d8047ba8dc4a909b67595b654d8e39 (127.5.176.194:41085)
I20260619 22:30:06.396531 25601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.194:37143
W20260619 22:30:06.503005 25924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:06.503293 25924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:06.503331 25924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:06.507824 25924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:06.507879 25924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:06.507936 25924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.176.195
I20260619 22:30:06.512739 25924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.176.195
--webserver_port=0
--tserver_master_addrs=127.5.176.254:43093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.176.195
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:06.513778 25924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:06.515249 25924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 22:30:06.525048 25924 server_base.cc:1034] running on GCE node
W20260619 22:30:06.525020 25929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:06.525007 25930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:06.525241 25932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:06.525707 25924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:06.526424 25924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:06.527613 25924 hybrid_clock.cc:648] HybridClock initialized: now 1781908206527555 us; error 68 us; skew 500 ppm
I20260619 22:30:06.529637 25924 webserver.cc:460] Webserver started at http://127.5.176.195:35783/ using document root <none> and password file <none>
I20260619 22:30:06.530308 25924 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:06.530401 25924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:06.530627 25924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:06.532645 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data/instance:
uuid: "db19f50eb8514f218fe724dfdbf6aa91"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "e77dd31dd568dbdbd412bf71d8038061"
server_key_iv: "cfd55d7a93631167c0ef3c9b62294d7f"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.533222 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance:
uuid: "db19f50eb8514f218fe724dfdbf6aa91"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "e77dd31dd568dbdbd412bf71d8038061"
server_key_iv: "cfd55d7a93631167c0ef3c9b62294d7f"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.536900 25924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 22:30:06.539290 25938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.540513 25924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:30:06.540656 25924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal
uuid: "db19f50eb8514f218fe724dfdbf6aa91"
format_stamp: "Formatted at 2026-06-19 22:30:06 on dist-test-slave-bt27"
server_key: "e77dd31dd568dbdbd412bf71d8038061"
server_key_iv: "cfd55d7a93631167c0ef3c9b62294d7f"
server_key_version: "encryptionkey@0"
I20260619 22:30:06.540776 25924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:06.550807 25924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:06.551446 25924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:06.551631 25924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:06.552186 25924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 22:30:06.553165 25924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 22:30:06.553236 25924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.553306 25924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 22:30:06.553344 25924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:06.562363 25924 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.195:44635
I20260619 22:30:06.562434 26051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.195:44635 every 8 connection(s)
I20260619 22:30:06.563366 25924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/data/info.pb
I20260619 22:30:06.566382 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 25924
I20260619 22:30:06.566514 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal/instance
I20260619 22:30:06.566880 5827 external_mini_cluster.cc:1428] Setting key cd57f937ff42f1f1fe38955bf229aa4b
I20260619 22:30:06.574065 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.565495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:54305 (local address 127.5.176.254:43093)
0619 22:30:06.565643 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.565648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.566791 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 22:30:06.568813 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.568816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.568871 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.568958 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:06.570093 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.570764 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.572190 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.572406 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.572491 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.573622 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:06.573625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:06.573627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:06.573640 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:06.573659 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:06.573667 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:06.573730 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.573893 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.573945 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:30:06.574961 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.565890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:43093 (local address 127.5.176.195:54305)
0619 22:30:06.566622 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.566676 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.568545 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.569160 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.569171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.569298 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:06.569958 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.569972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.570971 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.570974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.572082 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.572092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.572539 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.572542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.572690 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.573363 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 22:30:06.573364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:06.573405 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 22:30:06.573470 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:06.573744 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:06.573750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:06.573756 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.573856 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.574079 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260619 22:30:06.576088 26052 heartbeater.cc:347] Connected to a master server at 127.5.176.254:43093
I20260619 22:30:06.576316 26052 heartbeater.cc:464] Registering TS with master...
I20260619 22:30:06.576854 26052 heartbeater.cc:511] Master 127.5.176.254:43093 requested a full tablet report, sending...
I20260619 22:30:06.577915 25601 ts_manager.cc:194] Registered new tserver with Master: db19f50eb8514f218fe724dfdbf6aa91 (127.5.176.195:44635)
I20260619 22:30:06.578524 25601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.176.195:54305
I20260619 22:30:06.581287 5827 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 22:30:06.587064 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.583310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.5.176.254:43093)
0619 22:30:06.583437 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.583441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.583598 (+ 157us) server_negotiation.cc:407] Connection header received
0619 22:30:06.583729 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.583731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.583769 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.583838 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:06.584160 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.584666 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.585492 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.585696 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.585769 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.586632 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:06.586634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:06.586636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:06.586647 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:06.586664 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:06.586672 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:06.586731 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.586887 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.586939 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:30:06.589975 25601 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 22:30:06.593207 25601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 22:30:06.608670 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.602797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.5.176.195:44635)
0619 22:30:06.603219 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.603231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.603414 (+ 183us) server_negotiation.cc:407] Connection header received
0619 22:30:06.603486 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.603491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.603677 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.603840 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:06.604321 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.605476 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.607276 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.608082 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.608190 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.608353 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.608417 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.608515 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":138,"threads_started":1}
I20260619 22:30:06.608505 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.601861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:41085 (local address 127.0.0.1:51226)
0619 22:30:06.602825 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.602904 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.603027 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.603870 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.603876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.603900 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:06.604063 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.604069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.605618 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.605621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.606989 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.606999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.607902 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.607905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.608024 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.608072 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.608187 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.608321 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":14,"thread_start_us":99,"threads_started":1}
I20260619 22:30:06.608876 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.602052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.5.176.194:41085)
0619 22:30:06.603326 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.603348 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.603377 (+ 29us) server_negotiation.cc:407] Connection header received
0619 22:30:06.603462 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.603467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.603677 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.603845 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:06.604199 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.605481 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.607135 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.607782 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.607901 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.608112 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.608589 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.608699 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":146,"threads_started":1}
I20260619 22:30:06.609141 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.602174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.195:44635 (local address 127.0.0.1:37440)
0619 22:30:06.603247 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.603264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.603382 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.604011 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.604014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.604063 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:06.604197 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.604204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.605618 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.605621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.607141 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.607151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.608197 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.608201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.608291 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.608314 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.608392 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.608467 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":10,"spinlock_wait_cycles":54144,"thread_start_us":107,"threads_started":1}
I20260619 22:30:06.609787 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.602289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40171 (local address 127.0.0.1:34020)
0619 22:30:06.603342 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.603354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.603433 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.604011 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.604014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.604051 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:06.604197 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.604204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.606615 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.606618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.607692 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.607702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.609391 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.609398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.609496 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.609519 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.609580 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.609644 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260619 22:30:06.610404 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.602786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.5.176.193:40171)
0619 22:30:06.603326 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.603354 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.603507 (+ 153us) server_negotiation.cc:407] Connection header received
0619 22:30:06.603586 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.603590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.603727 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.603891 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:06.605387 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.606490 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.607840 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.608593 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.609906 (+ 1313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.610067 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.610108 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.610242 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":147,"threads_started":1}
I20260619 22:30:06.612480 25724 tablet_service.cc:1467] Processing CreateTablet for tablet 48403916a6b64cd9b2ba307aa30ed8fe (DEFAULT_TABLE table=test-table [id=989644acd3224334af04ef2378460134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:06.612485 25855 tablet_service.cc:1467] Processing CreateTablet for tablet 48403916a6b64cd9b2ba307aa30ed8fe (DEFAULT_TABLE table=test-table [id=989644acd3224334af04ef2378460134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:06.612486 25986 tablet_service.cc:1467] Processing CreateTablet for tablet 48403916a6b64cd9b2ba307aa30ed8fe (DEFAULT_TABLE table=test-table [id=989644acd3224334af04ef2378460134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 22:30:06.613483 25855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48403916a6b64cd9b2ba307aa30ed8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:06.613483 25986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48403916a6b64cd9b2ba307aa30ed8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:06.613490 25724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48403916a6b64cd9b2ba307aa30ed8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:06.619928 26069 tablet_bootstrap.cc:492] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f: Bootstrap starting.
I20260619 22:30:06.621295 26070 tablet_bootstrap.cc:492] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Bootstrap starting.
I20260619 22:30:06.622253 26069 tablet_bootstrap.cc:654] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:06.622504 26071 tablet_bootstrap.cc:492] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Bootstrap starting.
I20260619 22:30:06.623047 26069 log.cc:824] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:06.623370 26070 tablet_bootstrap.cc:654] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:06.624176 26070 log.cc:824] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:06.624681 26071 tablet_bootstrap.cc:654] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:06.625202 26069 tablet_bootstrap.cc:492] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f: No bootstrap required, opened a new log
I20260619 22:30:06.625414 26069 ts_tablet_manager.cc:1397] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 22:30:06.625519 26071 log.cc:824] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:06.626245 26070 tablet_bootstrap.cc:492] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: No bootstrap required, opened a new log
I20260619 22:30:06.626447 26070 ts_tablet_manager.cc:1397] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 22:30:06.627614 26071 tablet_bootstrap.cc:492] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: No bootstrap required, opened a new log
I20260619 22:30:06.627810 26071 ts_tablet_manager.cc:1397] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 22:30:06.629141 26069 raft_consensus.cc:348] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.629395 26069 raft_consensus.cc:374] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:06.629451 26069 raft_consensus.cc:729] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39439df93584f8dbef694e99f91546f, State: Initialized, Role: FOLLOWER
I20260619 22:30:06.629918 26069 consensus_queue.cc:260] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.630872 26069 ts_tablet_manager.cc:1428] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260619 22:30:06.630848 26070 raft_consensus.cc:348] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.631107 26070 raft_consensus.cc:374] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:06.631158 26070 raft_consensus.cc:729] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db19f50eb8514f218fe724dfdbf6aa91, State: Initialized, Role: FOLLOWER
I20260619 22:30:06.631256 25790 heartbeater.cc:503] Master 127.5.176.254:43093 was elected leader, sending a full tablet report...
I20260619 22:30:06.631590 26070 consensus_queue.cc:260] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.632403 26052 heartbeater.cc:503] Master 127.5.176.254:43093 was elected leader, sending a full tablet report...
I20260619 22:30:06.632560 26070 ts_tablet_manager.cc:1428] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 22:30:06.632946 26071 raft_consensus.cc:348] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.633196 26071 raft_consensus.cc:374] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:06.633263 26071 raft_consensus.cc:729] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d8047ba8dc4a909b67595b654d8e39, State: Initialized, Role: FOLLOWER
I20260619 22:30:06.633688 26071 consensus_queue.cc:260] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.634670 25921 heartbeater.cc:503] Master 127.5.176.254:43093 was elected leader, sending a full tablet report...
I20260619 22:30:06.634742 26071 ts_tablet_manager.cc:1428] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 22:30:06.636942 26076 raft_consensus.cc:482] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 22:30:06.637090 26076 raft_consensus.cc:504] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.638221 26076 leader_election.cc:288] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3d8047ba8dc4a909b67595b654d8e39 (127.5.176.194:41085), b39439df93584f8dbef694e99f91546f (127.5.176.193:40171)
I20260619 22:30:06.642848 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.638371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.194:41085 (local address 127.5.176.195:52075)
0619 22:30:06.638618 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.638633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.638717 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.639013 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.639019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.639035 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:06.639160 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.639169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.640175 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.640179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.641381 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.641395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.642384 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.642387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.642521 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.642544 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.642623 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.642716 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":23}
I20260619 22:30:06.642935 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.638540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:42675 (local address 127.5.176.193:40171)
0619 22:30:06.638645 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.638651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.638918 (+ 267us) server_negotiation.cc:407] Connection header received
0619 22:30:06.639024 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.639027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.639077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.639156 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:06.639446 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.640175 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.641639 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.642274 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.642378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.642419 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.642731 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.642807 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 22:30:06.643322 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.638672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.176.195:52075 (local address 127.5.176.194:41085)
0619 22:30:06.638780 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.638785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.638801 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:30:06.638848 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.638851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.638900 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.638991 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 22:30:06.639293 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.640072 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.641532 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.642121 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.642257 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.642308 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.642668 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.642838 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 22:30:06.643378 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.638465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40171 (local address 127.5.176.195:42675)
0619 22:30:06.638838 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.638850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:06.638930 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:06.639198 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:06.639203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:06.639225 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 22:30:06.639337 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.639341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.640277 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.640281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.641484 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:06.641500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.642479 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.642482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:06.642600 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.642627 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 22:30:06.643172 (+ 545us) client_negotiation.cc:240] Negotiation successful
0619 22:30:06.643249 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260619 22:30:06.643949 25875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48403916a6b64cd9b2ba307aa30ed8fe" candidate_uuid: "db19f50eb8514f218fe724dfdbf6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d8047ba8dc4a909b67595b654d8e39" is_pre_election: true
I20260619 22:30:06.643929 25744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48403916a6b64cd9b2ba307aa30ed8fe" candidate_uuid: "db19f50eb8514f218fe724dfdbf6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39439df93584f8dbef694e99f91546f" is_pre_election: true
I20260619 22:30:06.644239 25875 raft_consensus.cc:2454] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db19f50eb8514f218fe724dfdbf6aa91 in term 0.
I20260619 22:30:06.644253 25744 raft_consensus.cc:2454] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db19f50eb8514f218fe724dfdbf6aa91 in term 0.
I20260619 22:30:06.644738 25941 leader_election.cc:302] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b39439df93584f8dbef694e99f91546f, db19f50eb8514f218fe724dfdbf6aa91, e3d8047ba8dc4a909b67595b654d8e39; no voters:
I20260619 22:30:06.645021 26076 raft_consensus.cc:2781] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 22:30:06.645120 26076 raft_consensus.cc:482] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 22:30:06.645164 26076 raft_consensus.cc:3037] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:06.646188 26076 raft_consensus.cc:504] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.646528 26076 leader_election.cc:288] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [CANDIDATE]: Term 1 election: Requested vote from peers e3d8047ba8dc4a909b67595b654d8e39 (127.5.176.194:41085), b39439df93584f8dbef694e99f91546f (127.5.176.193:40171)
I20260619 22:30:06.646876 25744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48403916a6b64cd9b2ba307aa30ed8fe" candidate_uuid: "db19f50eb8514f218fe724dfdbf6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39439df93584f8dbef694e99f91546f"
I20260619 22:30:06.646894 25875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48403916a6b64cd9b2ba307aa30ed8fe" candidate_uuid: "db19f50eb8514f218fe724dfdbf6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d8047ba8dc4a909b67595b654d8e39"
I20260619 22:30:06.647003 25744 raft_consensus.cc:3037] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:06.647003 25875 raft_consensus.cc:3037] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:06.648252 25744 raft_consensus.cc:2454] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db19f50eb8514f218fe724dfdbf6aa91 in term 1.
I20260619 22:30:06.648252 25875 raft_consensus.cc:2454] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db19f50eb8514f218fe724dfdbf6aa91 in term 1.
I20260619 22:30:06.648541 25941 leader_election.cc:302] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b39439df93584f8dbef694e99f91546f, db19f50eb8514f218fe724dfdbf6aa91; no voters:
I20260619 22:30:06.648692 26076 raft_consensus.cc:2781] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:06.648938 26076 raft_consensus.cc:686] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 1 LEADER]: Becoming Leader. State: Replica: db19f50eb8514f218fe724dfdbf6aa91, State: Running, Role: LEADER
I20260619 22:30:06.649267 26076 consensus_queue.cc:237] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } }
I20260619 22:30:06.652474 25602 catalog_manager.cc:5495] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 reported cstate change: term changed from 0 to 1, leader changed from <none> to db19f50eb8514f218fe724dfdbf6aa91 (127.5.176.195). New cstate: current_term: 1 leader_uuid: "db19f50eb8514f218fe724dfdbf6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db19f50eb8514f218fe724dfdbf6aa91" member_type: VOTER last_known_addr { host: "127.5.176.195" port: 44635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260619 22:30:06.671779 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.667775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.5.176.195:44635)
0619 22:30:06.667915 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:06.667921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:06.667936 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:30:06.668144 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:06.668147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:06.668203 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:06.668314 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:06.668593 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.669170 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.670104 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:06.670385 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:06.670503 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:06.670854 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:06.671333 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:06.671476 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:06.671513 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 22:30:06.671567 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 22:30:06.677736 25875 raft_consensus.cc:1261] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 1 FOLLOWER]: Refusing update from remote peer db19f50eb8514f218fe724dfdbf6aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:06.677737 25744 raft_consensus.cc:1261] T 48403916a6b64cd9b2ba307aa30ed8fe P b39439df93584f8dbef694e99f91546f [term 1 FOLLOWER]: Refusing update from remote peer db19f50eb8514f218fe724dfdbf6aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:06.678483 26076 consensus_queue.cc:1035] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39439df93584f8dbef694e99f91546f" member_type: VOTER last_known_addr { host: "127.5.176.193" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:06.678772 26079 consensus_queue.cc:1035] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3d8047ba8dc4a909b67595b654d8e39" member_type: VOTER last_known_addr { host: "127.5.176.194" port: 41085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:06.696359 25602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:06.696573 25602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 22:30:06.699528 25602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8ac5b085cd94778b65e2a625d01386f: Sending DeleteTablet for 3 replicas of tablet 48403916a6b64cd9b2ba307aa30ed8fe
I20260619 22:30:06.700485 25855 tablet_service.cc:1514] Processing DeleteTablet for tablet 48403916a6b64cd9b2ba307aa30ed8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:06 UTC) from {username='slave'} at 127.0.0.1:51226
I20260619 22:30:06.700534 25724 tablet_service.cc:1514] Processing DeleteTablet for tablet 48403916a6b64cd9b2ba307aa30ed8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:06 UTC) from {username='slave'} at 127.0.0.1:34020
I20260619 22:30:06.700567 25986 tablet_service.cc:1514] Processing DeleteTablet for tablet 48403916a6b64cd9b2ba307aa30ed8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 22:30:06 UTC) from {username='slave'} at 127.0.0.1:37440
I20260619 22:30:06.701069 26091 tablet_replica.cc:330] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: stopping tablet replica
I20260619 22:30:06.701273 26093 tablet_replica.cc:330] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: stopping tablet replica
I20260619 22:30:06.701440 26091 raft_consensus.cc:2229] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 22:30:06.701571 26093 raft_consensus.cc:2229] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 22:30:06.701782 26091 raft_consensus.cc:2258] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:06.701936 26093 raft_consensus.cc:2258] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 22:30:06.703153 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25662
I20260619 22:30:06.703796 26091 ts_tablet_manager.cc:1905] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:06.704238 26093 ts_tablet_manager.cc:1905] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 22:30:06.706525 26091 ts_tablet_manager.cc:1918] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:06.706643 26091 log.cc:1196] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-1/wal/wals/48403916a6b64cd9b2ba307aa30ed8fe
I20260619 22:30:06.706928 26091 ts_tablet_manager.cc:1939] T 48403916a6b64cd9b2ba307aa30ed8fe P e3d8047ba8dc4a909b67595b654d8e39: Deleting consensus metadata
I20260619 22:30:06.706911 26093 ts_tablet_manager.cc:1918] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 22:30:06.707023 26093 log.cc:1196] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Deleting WAL directory at /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781907951519384-5827-0/minicluster-data/ts-2/wal/wals/48403916a6b64cd9b2ba307aa30ed8fe
I20260619 22:30:06.707372 26093 ts_tablet_manager.cc:1939] T 48403916a6b64cd9b2ba307aa30ed8fe P db19f50eb8514f218fe724dfdbf6aa91: Deleting consensus metadata
I20260619 22:30:06.708176 25587 catalog_manager.cc:4841] TS e3d8047ba8dc4a909b67595b654d8e39 (127.5.176.194:41085): tablet 48403916a6b64cd9b2ba307aa30ed8fe (table test-table [id=989644acd3224334af04ef2378460134]) successfully deleted
I20260619 22:30:06.708496 25589 catalog_manager.cc:4841] TS db19f50eb8514f218fe724dfdbf6aa91 (127.5.176.195:44635): tablet 48403916a6b64cd9b2ba307aa30ed8fe (table test-table [id=989644acd3224334af04ef2378460134]) successfully deleted
W20260619 22:30:06.712231 25588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.176.193:40171 (error 108)
I20260619 22:30:06.712447 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25793
I20260619 22:30:06.713567 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:06.713182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.193:40171 (local address 127.0.0.1:34024)
0619 22:30:06.713395 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:06.713470 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.193:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 22:30:06.713748 25588 catalog_manager.cc:4584] TS b39439df93584f8dbef694e99f91546f (127.5.176.193:40171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48403916a6b64cd9b2ba307aa30ed8fe: Network error: Client connection negotiation failed: client connection to 127.5.176.193:40171: connect: Connection refused (error 111)
I20260619 22:30:06.719578 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25924
I20260619 22:30:06.725410 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 25571
2026-06-19T22:30:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (979 ms)
[----------] 6 tests from EncryptionPolicyTest (10627 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:06 dist-test-slave-bt27 krb5kdc[26097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:06 dist-test-slave-bt27 krb5kdc[26097](info): set up 2 sockets
Jun 19 22:30:06 dist-test-slave-bt27 krb5kdc[26097](info): commencing operation
krb5kdc: starting...
W20260619 22:30:07.782414 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:07 dist-test-slave-bt27 krb5kdc[26097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:07.904721 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:08.015478 26110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:08.015771 26110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:08.015816 26110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:08.019963 26110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:08.020037 26110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:08.020057 26110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:08.020074 26110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:08.020090 26110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:08.025614 26110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:08.026814 26110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:08.027879 26110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:08.036665 26116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:08.036675 26115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:08.036871 26110 server_base.cc:1034] running on GCE node
W20260619 22:30:08.036665 26118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:08.037562 26110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:08.038813 26110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:08.040024 26110 hybrid_clock.cc:648] HybridClock initialized: now 1781908208040005 us; error 31 us; skew 500 ppm
Jun 19 22:30:08 dist-test-slave-bt27 krb5kdc[26097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:08.042984 26110 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:08.044240 26110 webserver.cc:460] Webserver started at http://127.0.0.1:34323/ using document root <none> and password file <none>
I20260619 22:30:08.044858 26110 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:08.044924 26110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:08.045163 26110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:08.047358 26110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "718a6e30cde34c33b38afc30503fbbc1"
format_stamp: "Formatted at 2026-06-19 22:30:08 on dist-test-slave-bt27"
server_key: "45ae54098ca7f4f60263225a393d579b"
server_key_iv: "90bff7e79c603236af72acae0c29a715"
server_key_version: "encryptionkey@0"
I20260619 22:30:08.047943 26110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "718a6e30cde34c33b38afc30503fbbc1"
format_stamp: "Formatted at 2026-06-19 22:30:08 on dist-test-slave-bt27"
server_key: "45ae54098ca7f4f60263225a393d579b"
server_key_iv: "90bff7e79c603236af72acae0c29a715"
server_key_version: "encryptionkey@0"
I20260619 22:30:08.051862 26110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:08.054432 26125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:08.055727 26110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:08.055871 26110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "718a6e30cde34c33b38afc30503fbbc1"
format_stamp: "Formatted at 2026-06-19 22:30:08 on dist-test-slave-bt27"
server_key: "45ae54098ca7f4f60263225a393d579b"
server_key_iv: "90bff7e79c603236af72acae0c29a715"
server_key_version: "encryptionkey@0"
I20260619 22:30:08.055990 26110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:08.071805 26110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:08.075280 26110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:08.075498 26110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:08.083309 26110 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42985
I20260619 22:30:08.083343 26177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42985 every 8 connection(s)
I20260619 22:30:08.084312 26110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:08.087313 26178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:08.091421 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26110
I20260619 22:30:08.091591 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:08.091928 5827 external_mini_cluster.cc:1428] Setting key 6f847e23a68ddedc2849087013177db1
I20260619 22:30:08.093475 26178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1: Bootstrap starting.
I20260619 22:30:08.096153 26178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:08.096922 26178 log.cc:824] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:08.099000 26178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1: No bootstrap required, opened a new log
Jun 19 22:30:08 dist-test-slave-bt27 krb5kdc[26097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:30:08.103015 26178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718a6e30cde34c33b38afc30503fbbc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260619 22:30:08.103282 26178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:08.103374 26178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718a6e30cde34c33b38afc30503fbbc1, State: Initialized, Role: FOLLOWER
I20260619 22:30:08.103873 26178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718a6e30cde34c33b38afc30503fbbc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260619 22:30:08.104033 26178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:08.104123 26178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:08.104223 26178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:08.105180 26178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718a6e30cde34c33b38afc30503fbbc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260619 22:30:08.105562 26178 leader_election.cc:302] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 718a6e30cde34c33b38afc30503fbbc1; no voters:
I20260619 22:30:08.105927 26178 leader_election.cc:288] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:08.106038 26183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:08.106285 26183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [term 1 LEADER]: Becoming Leader. State: Replica: 718a6e30cde34c33b38afc30503fbbc1, State: Running, Role: LEADER
I20260619 22:30:08.106604 26183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718a6e30cde34c33b38afc30503fbbc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } }
I20260619 22:30:08.106827 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:08.093311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50781 (local address 127.0.0.1:42985)
0619 22:30:08.093775 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:08.093789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:08.093824 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:08.094535 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:08.094551 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:08.094895 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:08.095171 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:08.096199 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:08.097382 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:08.098343 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:08.098823 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:08.098995 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:08.101562 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:08.101567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:08.101582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:08.101610 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:08.103489 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:08.104013 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:08.104022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:08.104028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:08.104186 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:08.104530 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:08.104533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:08.104534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:08.104932 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:08.105154 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:08.105472 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 22:30:08.105734 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":147,"threads_started":1}
I20260619 22:30:08.107328 26178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:08.107872 26183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 718a6e30cde34c33b38afc30503fbbc1. Latest consensus state: current_term: 1 leader_uuid: "718a6e30cde34c33b38afc30503fbbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718a6e30cde34c33b38afc30503fbbc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } } }
I20260619 22:30:08.107982 26183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:08.108357 26184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "718a6e30cde34c33b38afc30503fbbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718a6e30cde34c33b38afc30503fbbc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42985 } } }
I20260619 22:30:08.108484 26184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:08.109108 26187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:08.110198 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:08.112767 26187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:08.118741 26187 catalog_manager.cc:1347] Generated new cluster ID: 54c8f00e83f848b8ae6d769488bd4a6e
I20260619 22:30:08.118818 26187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:08.123368 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:08.112417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34627 (local address 127.0.0.1:42985)
0619 22:30:08.112544 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:08.112548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:08.114464 (+ 1916us) server_negotiation.cc:407] Connection header received
0619 22:30:08.114639 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:08.114642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:08.114699 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:08.114755 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:08.115740 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:08.116469 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:08.117483 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:08.117739 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:08.117856 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:08.119656 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:08.119660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:08.119663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:08.119694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:08.121401 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:08.121945 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:08.121949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:08.121951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:08.122027 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:08.122410 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:08.122413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:08.122415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:08.122612 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:08.122721 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:08.123098 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 22:30:08.123203 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:30:08.177872 26187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:08.179219 26187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:08.185992 26187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 718a6e30cde34c33b38afc30503fbbc1: Generated new TSK 0
I20260619 22:30:08.186801 26187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:08.192327 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26110
W20260619 22:30:08.197992 26095 connection.cc:535] client connection to 127.0.0.1:42985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42985): Connection reset by peer (error 104)
2026-06-19T22:30:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:08 dist-test-slave-bt27 krb5kdc[26209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:08 dist-test-slave-bt27 krb5kdc[26209](info): set up 2 sockets
Jun 19 22:30:08 dist-test-slave-bt27 krb5kdc[26209](info): commencing operation
krb5kdc: starting...
W20260619 22:30:09.238399 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:09 dist-test-slave-bt27 krb5kdc[26209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:09.364730 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:09.476138 26222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:09.476464 26222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:09.476545 26222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:09.480613 26222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:09.480724 26222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:09.480763 26222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:09.480803 26222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:09.480836 26222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:09.486208 26222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:09.487318 26222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:09.488349 26222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:09.496800 26230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:09.496779 26228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:09.496809 26227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:09.497087 26222 server_base.cc:1034] running on GCE node
I20260619 22:30:09.497789 26222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:09.498925 26222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:09.500116 26222 hybrid_clock.cc:648] HybridClock initialized: now 1781908209500087 us; error 44 us; skew 500 ppm
Jun 19 22:30:09 dist-test-slave-bt27 krb5kdc[26209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:09.502918 26222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:09.503906 26222 webserver.cc:460] Webserver started at http://127.0.0.1:33459/ using document root <none> and password file <none>
I20260619 22:30:09.505009 26222 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:09.505131 26222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:09.505383 26222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:09.507696 26222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b"
format_stamp: "Formatted at 2026-06-19 22:30:09 on dist-test-slave-bt27"
server_key: "2bb909ee2e12de31c6479dee81414be7"
server_key_iv: "28a64e8c5da2df8c80a7d44db783f06b"
server_key_version: "encryptionkey@0"
I20260619 22:30:09.508325 26222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b"
format_stamp: "Formatted at 2026-06-19 22:30:09 on dist-test-slave-bt27"
server_key: "2bb909ee2e12de31c6479dee81414be7"
server_key_iv: "28a64e8c5da2df8c80a7d44db783f06b"
server_key_version: "encryptionkey@0"
I20260619 22:30:09.512639 26222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:09.515404 26237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:09.516785 26222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:30:09.516968 26222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b"
format_stamp: "Formatted at 2026-06-19 22:30:09 on dist-test-slave-bt27"
server_key: "2bb909ee2e12de31c6479dee81414be7"
server_key_iv: "28a64e8c5da2df8c80a7d44db783f06b"
server_key_version: "encryptionkey@0"
I20260619 22:30:09.517102 26222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:09.530792 26222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:09.533660 26222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:09.533878 26222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:09.541327 26222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41227
I20260619 22:30:09.541334 26289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41227 every 8 connection(s)
I20260619 22:30:09.542452 26222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:09.545480 26290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:09.551090 26290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b: Bootstrap starting.
I20260619 22:30:09.551636 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26222
I20260619 22:30:09.551779 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:09.552070 5827 external_mini_cluster.cc:1428] Setting key 019323c40438f41bec6db7c4ab6b61cd
I20260619 22:30:09.553834 26290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:09.554762 26290 log.cc:824] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:09.556768 26290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b: No bootstrap required, opened a new log
I20260619 22:30:09.561030 26290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41227 } }
I20260619 22:30:09.561352 26290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:09.561410 26290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434fd3f1bdbf4d90ab183b8a6ddfe17b, State: Initialized, Role: FOLLOWER
I20260619 22:30:09.561882 26290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41227 } }
I20260619 22:30:09.562038 26290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:09.562125 26290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:09.562263 26290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:09.562443 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:09.554284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39063 (local address 127.0.0.1:41227)
0619 22:30:09.554734 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:09.554750 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:09.554785 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:09.555517 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:09.555533 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:09.555862 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:09.556141 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:09.556510 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:09.557531 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:09.558842 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:09.559307 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:09.559425 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:09.560439 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:09.560444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:09.560466 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:09.560858 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:09.560912 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:09.560935 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:09.561038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:09.561254 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 22:30:09.561426 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":151,"threads_started":1}
I20260619 22:30:09.563462 26290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41227 } }
I20260619 22:30:09.563800 26290 leader_election.cc:302] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 434fd3f1bdbf4d90ab183b8a6ddfe17b; no voters:
I20260619 22:30:09.564157 26290 leader_election.cc:288] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:09.564224 26295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:09.564975 26295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [term 1 LEADER]: Becoming Leader. State: Replica: 434fd3f1bdbf4d90ab183b8a6ddfe17b, State: Running, Role: LEADER
I20260619 22:30:09.565270 26295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41227 } }
I20260619 22:30:09.565507 26290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:09.565639 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:09.567385 26297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 434fd3f1bdbf4d90ab183b8a6ddfe17b. Latest consensus state: current_term: 1 leader_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41227 } } }
I20260619 22:30:09.567664 26297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:09.567783 26296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434fd3f1bdbf4d90ab183b8a6ddfe17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41227 } } }
I20260619 22:30:09.568095 26296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b [sys.catalog]: This master's current role is: LEADER
W20260619 22:30:09.570459 26313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:09.570538 26313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:09.570657 26314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:09.571599 26314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:09.575451 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:09.569014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47629 (local address 127.0.0.1:41227)
0619 22:30:09.569144 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:09.569148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:09.570441 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 22:30:09.570660 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:09.570663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:09.570718 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:09.570795 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:09.572085 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:09.572611 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:09.573447 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:09.573791 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:09.573896 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:09.574949 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:09.574952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:09.574955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:09.574970 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:09.574999 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:09.575012 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:09.575106 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:09.575280 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 22:30:09.575318 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 22:30:09.577824 26314 catalog_manager.cc:1347] Generated new cluster ID: 8f3f0d0a749741569d9c5098ad03aa09
I20260619 22:30:09.577921 26314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:09.589725 26314 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:09.590821 26314 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:09.601917 26314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 434fd3f1bdbf4d90ab183b8a6ddfe17b: Generated new TSK 0
I20260619 22:30:09.602747 26314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:09.612574 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26222
W20260619 22:30:09.618418 26207 connection.cc:535] client connection to 127.0.0.1:41227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41227): Connection reset by peer (error 104)
2026-06-19T22:30:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:09 dist-test-slave-bt27 krb5kdc[26321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:09 dist-test-slave-bt27 krb5kdc[26321](info): set up 2 sockets
Jun 19 22:30:09 dist-test-slave-bt27 krb5kdc[26321](info): commencing operation
krb5kdc: starting...
W20260619 22:30:10.670387 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:10 dist-test-slave-bt27 krb5kdc[26321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:10.793931 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:10.905184 26334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:10.905472 26334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:10.905517 26334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:10.909660 26334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:10.909732 26334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:10.909754 26334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:10.909771 26334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:10.909787 26334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:10.915215 26334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:10.916328 26334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:10.917330 26334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:10.926182 26339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:10.926222 26342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:10.926348 26334 server_base.cc:1034] running on GCE node
W20260619 22:30:10.926224 26340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:10.926988 26334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:10.928156 26334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:10.929358 26334 hybrid_clock.cc:648] HybridClock initialized: now 1781908210929326 us; error 43 us; skew 500 ppm
Jun 19 22:30:10 dist-test-slave-bt27 krb5kdc[26321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:10.932160 26334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:10.933199 26334 webserver.cc:460] Webserver started at http://127.0.0.1:41613/ using document root <none> and password file <none>
I20260619 22:30:10.933808 26334 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:10.933900 26334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:10.934173 26334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:10.936264 26334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "2730a9b338a9458888aafd67224bcae3"
format_stamp: "Formatted at 2026-06-19 22:30:10 on dist-test-slave-bt27"
server_key: "cba7b12c9397d1aa09920ac889db71d1"
server_key_iv: "e37fb276e7be99ad09f150a7c4f0924c"
server_key_version: "encryptionkey@0"
I20260619 22:30:10.936825 26334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "2730a9b338a9458888aafd67224bcae3"
format_stamp: "Formatted at 2026-06-19 22:30:10 on dist-test-slave-bt27"
server_key: "cba7b12c9397d1aa09920ac889db71d1"
server_key_iv: "e37fb276e7be99ad09f150a7c4f0924c"
server_key_version: "encryptionkey@0"
I20260619 22:30:10.940570 26334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 22:30:10.943143 26349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:10.944420 26334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:10.944586 26334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "2730a9b338a9458888aafd67224bcae3"
format_stamp: "Formatted at 2026-06-19 22:30:10 on dist-test-slave-bt27"
server_key: "cba7b12c9397d1aa09920ac889db71d1"
server_key_iv: "e37fb276e7be99ad09f150a7c4f0924c"
server_key_version: "encryptionkey@0"
I20260619 22:30:10.944705 26334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:10.949335 26334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:10.952527 26334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:10.952721 26334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:10.959766 26334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36235
I20260619 22:30:10.959781 26401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36235 every 8 connection(s)
I20260619 22:30:10.960655 26334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:10.963343 26402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:10.968701 26402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3: Bootstrap starting.
I20260619 22:30:10.970363 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26334
I20260619 22:30:10.970508 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:10.970789 5827 external_mini_cluster.cc:1428] Setting key e18d9b06b9bdfb8023b820e2a3f15bfb
I20260619 22:30:10.971413 26402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:10.972302 26402 log.cc:824] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:10.974424 26402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3: No bootstrap required, opened a new log
I20260619 22:30:10.975996 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:10.971957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45129 (local address 127.0.0.1:36235)
0619 22:30:10.972465 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:10.972486 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:10.972530 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:30:10.973375 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:10.973394 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:10.973672 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:10.974016 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:10.974846 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:10.974851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:10.974861 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:10.974871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:10.974922 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:10.974941 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:10.975060 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:10.975155 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 22:30:10.975311 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":131,"threads_started":1}
I20260619 22:30:10.978705 26402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2730a9b338a9458888aafd67224bcae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36235 } }
I20260619 22:30:10.979036 26402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:10.979101 26402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2730a9b338a9458888aafd67224bcae3, State: Initialized, Role: FOLLOWER
I20260619 22:30:10.979115 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:10.979503 26402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2730a9b338a9458888aafd67224bcae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36235 } }
I20260619 22:30:10.979621 26402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:10.979705 26402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:10.979801 26402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:10.980815 26402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2730a9b338a9458888aafd67224bcae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36235 } }
I20260619 22:30:10.981158 26402 leader_election.cc:302] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2730a9b338a9458888aafd67224bcae3; no voters:
I20260619 22:30:10.981518 26402 leader_election.cc:288] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:10.981602 26412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:10.981827 26412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [term 1 LEADER]: Becoming Leader. State: Replica: 2730a9b338a9458888aafd67224bcae3, State: Running, Role: LEADER
I20260619 22:30:10.982201 26412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2730a9b338a9458888aafd67224bcae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36235 } }
I20260619 22:30:10.982957 26402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:10.983251 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:10.981276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52895 (local address 127.0.0.1:36235)
0619 22:30:10.981429 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:10.981434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:10.981726 (+ 292us) server_negotiation.cc:407] Connection header received
0619 22:30:10.981771 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:10.981774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:10.981807 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:10.981874 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:10.982714 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:10.982717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:10.982719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:10.982730 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:10.982756 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:10.982769 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:10.982840 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:10.983106 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 22:30:10.983143 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 22:30:10.983951 26413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2730a9b338a9458888aafd67224bcae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2730a9b338a9458888aafd67224bcae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36235 } } }
I20260619 22:30:10.984084 26413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:10.984508 26421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:10.984659 26414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2730a9b338a9458888aafd67224bcae3. Latest consensus state: current_term: 1 leader_uuid: "2730a9b338a9458888aafd67224bcae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2730a9b338a9458888aafd67224bcae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36235 } } }
I20260619 22:30:10.984846 26414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:10.987726 26421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:10.992763 26421 catalog_manager.cc:1347] Generated new cluster ID: ca6a9871f53a432db66a9f2b35066d90
I20260619 22:30:10.992828 26421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:11.006585 26421 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:11.006959 26421 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:11.011313 26421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2730a9b338a9458888aafd67224bcae3: Generated new TSK 0
I20260619 22:30:11.012079 26421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:11.016165 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26334
2026-06-19T22:30:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:11 dist-test-slave-bt27 krb5kdc[26433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:11 dist-test-slave-bt27 krb5kdc[26433](info): set up 2 sockets
Jun 19 22:30:11 dist-test-slave-bt27 krb5kdc[26433](info): commencing operation
krb5kdc: starting...
W20260619 22:30:12.066507 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:12 dist-test-slave-bt27 krb5kdc[26433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:12.189772 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:12.299643 26446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:12.299976 26446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:12.300055 26446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:12.304291 26446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:12.304386 26446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:12.304425 26446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:12.304479 26446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:12.304517 26446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:12.310101 26446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:12.311287 26446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:12.312363 26446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:12.321203 26451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:12.321377 26454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:12.321369 26446 server_base.cc:1034] running on GCE node
W20260619 22:30:12.321210 26452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:12.322212 26446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:12.323427 26446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:12.324661 26446 hybrid_clock.cc:648] HybridClock initialized: now 1781908212324606 us; error 66 us; skew 500 ppm
Jun 19 22:30:12 dist-test-slave-bt27 krb5kdc[26433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:12.331238 26446 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:12.332273 26446 webserver.cc:460] Webserver started at http://127.0.0.1:45361/ using document root <none> and password file <none>
I20260619 22:30:12.332885 26446 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:12.332978 26446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:12.333212 26446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:12.335347 26446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "668deaa72b544d5aa884bd9f1685e84f"
format_stamp: "Formatted at 2026-06-19 22:30:12 on dist-test-slave-bt27"
server_key: "3e43f94d70646396313a5d97573e3d62"
server_key_iv: "4577742a6da122d4d660ff66fd85777c"
server_key_version: "encryptionkey@0"
I20260619 22:30:12.335927 26446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "668deaa72b544d5aa884bd9f1685e84f"
format_stamp: "Formatted at 2026-06-19 22:30:12 on dist-test-slave-bt27"
server_key: "3e43f94d70646396313a5d97573e3d62"
server_key_iv: "4577742a6da122d4d660ff66fd85777c"
server_key_version: "encryptionkey@0"
I20260619 22:30:12.339754 26446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 22:30:12.342301 26461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:12.343612 26446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:12.343776 26446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "668deaa72b544d5aa884bd9f1685e84f"
format_stamp: "Formatted at 2026-06-19 22:30:12 on dist-test-slave-bt27"
server_key: "3e43f94d70646396313a5d97573e3d62"
server_key_iv: "4577742a6da122d4d660ff66fd85777c"
server_key_version: "encryptionkey@0"
I20260619 22:30:12.343905 26446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:12.360549 26446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:12.363910 26446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:12.364145 26446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:12.372460 26446 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41533
I20260619 22:30:12.372474 26513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41533 every 8 connection(s)
I20260619 22:30:12.373479 26446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:12.376582 26514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:12.376765 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26446
I20260619 22:30:12.376936 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:12.377223 5827 external_mini_cluster.cc:1428] Setting key 1469d3675a4e49bc1b1077bd7d141748
I20260619 22:30:12.383876 26514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f: Bootstrap starting.
Jun 19 22:30:12 dist-test-slave-bt27 krb5kdc[26433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:30:12.386765 26514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:12.387538 26514 log.cc:824] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:12.389889 26514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f: No bootstrap required, opened a new log
I20260619 22:30:12.392709 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:12.378647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43767 (local address 127.0.0.1:41533)
0619 22:30:12.379092 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:12.379107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:12.379144 (+ 37us) server_negotiation.cc:407] Connection header received
0619 22:30:12.379944 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:12.379963 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:12.380324 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:12.380622 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:12.381568 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:12.382627 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:12.383808 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:12.384368 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:12.384507 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:12.387380 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:12.387385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:12.387401 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:12.387430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:12.389385 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:12.389827 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:12.389832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:12.389838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:12.389980 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:12.390318 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:12.390323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:12.390326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:12.390867 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:12.391070 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:12.391295 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 22:30:12.391616 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":154,"threads_started":1}
I20260619 22:30:12.394629 26514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668deaa72b544d5aa884bd9f1685e84f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260619 22:30:12.394914 26514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:12.395015 26514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668deaa72b544d5aa884bd9f1685e84f, State: Initialized, Role: FOLLOWER
I20260619 22:30:12.395490 26514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668deaa72b544d5aa884bd9f1685e84f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260619 22:30:12.395648 26514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:12.395718 26514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:12.395819 26514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:12.395853 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:12.396788 26514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668deaa72b544d5aa884bd9f1685e84f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260619 22:30:12.397224 26514 leader_election.cc:302] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 668deaa72b544d5aa884bd9f1685e84f; no voters:
I20260619 22:30:12.397581 26514 leader_election.cc:288] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:12.397667 26523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:12.397841 26523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [term 1 LEADER]: Becoming Leader. State: Replica: 668deaa72b544d5aa884bd9f1685e84f, State: Running, Role: LEADER
I20260619 22:30:12.398355 26523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668deaa72b544d5aa884bd9f1685e84f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260619 22:30:12.398967 26514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:12.400157 26524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "668deaa72b544d5aa884bd9f1685e84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668deaa72b544d5aa884bd9f1685e84f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } } }
I20260619 22:30:12.400290 26524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:12.400671 26525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 668deaa72b544d5aa884bd9f1685e84f. Latest consensus state: current_term: 1 leader_uuid: "668deaa72b544d5aa884bd9f1685e84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668deaa72b544d5aa884bd9f1685e84f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } } }
I20260619 22:30:12.401315 26525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:12.401577 26533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:12.405668 26533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:12.407485 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:12.398006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47563 (local address 127.0.0.1:41533)
0619 22:30:12.398114 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:12.398119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:12.398484 (+ 365us) server_negotiation.cc:407] Connection header received
0619 22:30:12.398663 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:12.398666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:12.398712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:12.398813 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:12.399609 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:12.400392 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:12.401511 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:12.401859 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:12.401949 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:12.403705 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:12.403708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:12.403711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:12.403737 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:12.405512 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:12.406015 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:12.406019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:12.406021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:12.406105 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:12.406481 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:12.406484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:12.406485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:12.406711 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:12.406811 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:12.407168 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 22:30:12.407309 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 22:30:12.412058 26533 catalog_manager.cc:1347] Generated new cluster ID: 3a6ab321080f418182bb33e24d98a185
I20260619 22:30:12.412137 26533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:12.436770 26533 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:12.438253 26533 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:12.455355 26533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 668deaa72b544d5aa884bd9f1685e84f: Generated new TSK 0
I20260619 22:30:12.456275 26533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:12.471830 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26446
2026-06-19T22:30:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:12 dist-test-slave-bt27 krb5kdc[26545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:12 dist-test-slave-bt27 krb5kdc[26545](info): set up 2 sockets
Jun 19 22:30:12 dist-test-slave-bt27 krb5kdc[26545](info): commencing operation
krb5kdc: starting...
W20260619 22:30:13.518652 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:13 dist-test-slave-bt27 krb5kdc[26545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:13.641185 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:13.753469 26558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:13.753769 26558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:13.753813 26558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:13.757938 26558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:13.758003 26558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:13.758023 26558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:13.758040 26558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:13.758109 26558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:13.763651 26558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:13.764786 26558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:13.765820 26558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:13.774598 26564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:13.774611 26566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:13.774598 26563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:13.774892 26558 server_base.cc:1034] running on GCE node
I20260619 22:30:13.775606 26558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:13.776747 26558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:13.777937 26558 hybrid_clock.cc:648] HybridClock initialized: now 1781908213777920 us; error 31 us; skew 500 ppm
Jun 19 22:30:13 dist-test-slave-bt27 krb5kdc[26545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:13.780854 26558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:13.781728 26558 webserver.cc:460] Webserver started at http://127.0.0.1:36933/ using document root <none> and password file <none>
I20260619 22:30:13.782339 26558 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:13.782402 26558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:13.782593 26558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:13.784613 26558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "3a67d405d7ef4dc3b75940164e8f0c0f"
format_stamp: "Formatted at 2026-06-19 22:30:13 on dist-test-slave-bt27"
server_key: "84c103bed31c8e3bccd632337a4fffce"
server_key_iv: "9e29efa9fda94d909c53a19f2b79d292"
server_key_version: "encryptionkey@0"
I20260619 22:30:13.785127 26558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "3a67d405d7ef4dc3b75940164e8f0c0f"
format_stamp: "Formatted at 2026-06-19 22:30:13 on dist-test-slave-bt27"
server_key: "84c103bed31c8e3bccd632337a4fffce"
server_key_iv: "9e29efa9fda94d909c53a19f2b79d292"
server_key_version: "encryptionkey@0"
I20260619 22:30:13.788849 26558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 22:30:13.791327 26573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:13.792676 26558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:13.792851 26558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "3a67d405d7ef4dc3b75940164e8f0c0f"
format_stamp: "Formatted at 2026-06-19 22:30:13 on dist-test-slave-bt27"
server_key: "84c103bed31c8e3bccd632337a4fffce"
server_key_iv: "9e29efa9fda94d909c53a19f2b79d292"
server_key_version: "encryptionkey@0"
I20260619 22:30:13.792985 26558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:13.804540 26558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:13.807677 26558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:13.807893 26558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:13.815611 26558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43245
I20260619 22:30:13.815639 26625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43245 every 8 connection(s)
I20260619 22:30:13.816704 26558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:13.818879 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26558
I20260619 22:30:13.819031 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:13.819341 5827 external_mini_cluster.cc:1428] Setting key aeeb2994f936a411e6fc18195065d5e4
I20260619 22:30:13.819711 26626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:13.825883 26626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f: Bootstrap starting.
I20260619 22:30:13.828238 26626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:13.828287 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:13.820792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54101 (local address 127.0.0.1:43245)
0619 22:30:13.821240 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:13.821257 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:13.821292 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:13.821943 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:13.821958 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:13.822366 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:13.822699 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:13.823121 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:13.824086 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:13.825025 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:13.825492 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:13.825661 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:13.826627 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:13.826632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:13.826646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:13.827038 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:13.827092 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:13.827115 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:13.827229 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:13.827353 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 22:30:13.827578 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260619 22:30:13.829128 26626 log.cc:824] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:13.831269 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:13.832122 26626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f: No bootstrap required, opened a new log
I20260619 22:30:13.836370 26626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260619 22:30:13.836699 26626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:13.836751 26626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a67d405d7ef4dc3b75940164e8f0c0f, State: Initialized, Role: FOLLOWER
I20260619 22:30:13.837215 26626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260619 22:30:13.837406 26626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:13.837481 26626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:13.837594 26626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:13.838300 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:13.833434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56635 (local address 127.0.0.1:43245)
0619 22:30:13.833548 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:13.833553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:13.833667 (+ 114us) server_negotiation.cc:407] Connection header received
0619 22:30:13.833765 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:13.833769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:13.833823 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:13.833907 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:13.834292 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:13.835591 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:13.836496 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:13.836856 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:13.836963 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:13.837865 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:13.837872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:13.837878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:13.837893 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:13.837920 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:13.837936 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:13.838017 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:13.838088 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 22:30:13.838164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 22:30:13.838876 26626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260619 22:30:13.839234 26626 leader_election.cc:302] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a67d405d7ef4dc3b75940164e8f0c0f; no voters:
I20260619 22:30:13.839599 26626 leader_election.cc:288] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:13.839711 26636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:13.839946 26636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [term 1 LEADER]: Becoming Leader. State: Replica: 3a67d405d7ef4dc3b75940164e8f0c0f, State: Running, Role: LEADER
I20260619 22:30:13.840309 26636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260619 22:30:13.840927 26626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:13.842016 26637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } } }
I20260619 22:30:13.842167 26637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:13.842454 26638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a67d405d7ef4dc3b75940164e8f0c0f. Latest consensus state: current_term: 1 leader_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67d405d7ef4dc3b75940164e8f0c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } } }
I20260619 22:30:13.842537 26638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:13.842552 26645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:13.845405 26645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:13.850939 26645 catalog_manager.cc:1347] Generated new cluster ID: 9fb0d426d975404f8fe3bf6ef3a8e3ec
I20260619 22:30:13.851023 26645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:13.856938 26645 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:13.858218 26645 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:13.872169 26645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a67d405d7ef4dc3b75940164e8f0c0f: Generated new TSK 0
I20260619 22:30:13.873121 26645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:13.880892 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26558
2026-06-19T22:30:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:13 dist-test-slave-bt27 krb5kdc[26657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:13 dist-test-slave-bt27 krb5kdc[26657](info): set up 2 sockets
Jun 19 22:30:13 dist-test-slave-bt27 krb5kdc[26657](info): commencing operation
krb5kdc: starting...
W20260619 22:30:14.922708 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:15 dist-test-slave-bt27 krb5kdc[26657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:15.043467 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:15.154770 26670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:15.155138 26670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:15.155225 26670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:15.159504 26670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:15.159621 26670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:15.159660 26670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:15.159710 26670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:15.159746 26670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:15.165364 26670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:15.166601 26670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:15.167673 26670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:15.177276 26675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:15.177276 26678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:15.177330 26670 server_base.cc:1034] running on GCE node
W20260619 22:30:15.177276 26676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:15.178329 26670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:15.179553 26670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:15.180737 26670 hybrid_clock.cc:648] HybridClock initialized: now 1781908215180713 us; error 50 us; skew 500 ppm
Jun 19 22:30:15 dist-test-slave-bt27 krb5kdc[26657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:15.183777 26670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:15.184785 26670 webserver.cc:460] Webserver started at http://127.0.0.1:40921/ using document root <none> and password file <none>
I20260619 22:30:15.185333 26670 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:15.185395 26670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:15.185563 26670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:15.187747 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17"
format_stamp: "Formatted at 2026-06-19 22:30:15 on dist-test-slave-bt27"
server_key: "aed1dc969f07119e3fcaf8cb7f30bafb"
server_key_iv: "c9eaba5453dc4bc8684710758734f482"
server_key_version: "encryptionkey@0"
I20260619 22:30:15.188308 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17"
format_stamp: "Formatted at 2026-06-19 22:30:15 on dist-test-slave-bt27"
server_key: "aed1dc969f07119e3fcaf8cb7f30bafb"
server_key_iv: "c9eaba5453dc4bc8684710758734f482"
server_key_version: "encryptionkey@0"
I20260619 22:30:15.192219 26670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:15.194840 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:15.196235 26670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:15.196405 26670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17"
format_stamp: "Formatted at 2026-06-19 22:30:15 on dist-test-slave-bt27"
server_key: "aed1dc969f07119e3fcaf8cb7f30bafb"
server_key_iv: "c9eaba5453dc4bc8684710758734f482"
server_key_version: "encryptionkey@0"
I20260619 22:30:15.196542 26670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:15.201669 26670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:15.205036 26670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:15.205245 26670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:15.212697 26670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46817
I20260619 22:30:15.212723 26737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46817 every 8 connection(s)
I20260619 22:30:15.213742 26670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:15.216952 26738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:15.220146 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26670
I20260619 22:30:15.220268 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:15.220564 5827 external_mini_cluster.cc:1428] Setting key 84fbf6bcb52d3bb415e0d2e1551a90d1
I20260619 22:30:15.224252 26738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17: Bootstrap starting.
I20260619 22:30:15.226791 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:15.222202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53127 (local address 127.0.0.1:46817)
0619 22:30:15.222670 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:15.222694 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:15.222744 (+ 50us) server_negotiation.cc:407] Connection header received
0619 22:30:15.223665 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:15.223687 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:15.224067 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:15.224479 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:15.225272 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:15.225278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:15.225292 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:15.225304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:15.225372 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:15.225403 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:15.225552 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:15.225633 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 22:30:15.225805 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260619 22:30:15.227288 26738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:15.228191 26738 log.cc:824] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:15.229805 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:15.230705 26738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17: No bootstrap required, opened a new log
I20260619 22:30:15.233771 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:15.232143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44193 (local address 127.0.0.1:46817)
0619 22:30:15.232268 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:15.232272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:15.232432 (+ 160us) server_negotiation.cc:407] Connection header received
0619 22:30:15.232522 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:15.232525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:15.232559 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:15.232648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:15.233400 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:15.233404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:15.233408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:15.233420 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:15.233450 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:15.233463 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:15.233552 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:15.233604 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 22:30:15.233642 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 22:30:15.235419 26738 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260619 22:30:15.235654 26738 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:15.235719 26738 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2528f951cfb4cc4ac5cd8ab0cec4b17, State: Initialized, Role: FOLLOWER
I20260619 22:30:15.236150 26738 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260619 22:30:15.236289 26738 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:15.236356 26738 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:15.236467 26738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:15.237475 26738 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260619 22:30:15.237843 26738 leader_election.cc:302] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2528f951cfb4cc4ac5cd8ab0cec4b17; no voters:
I20260619 22:30:15.238191 26738 leader_election.cc:288] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:15.238341 26748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:15.238559 26748 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [term 1 LEADER]: Becoming Leader. State: Replica: a2528f951cfb4cc4ac5cd8ab0cec4b17, State: Running, Role: LEADER
I20260619 22:30:15.238916 26748 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260619 22:30:15.239529 26738 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:15.240646 26750 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2528f951cfb4cc4ac5cd8ab0cec4b17. Latest consensus state: current_term: 1 leader_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } } }
I20260619 22:30:15.240792 26750 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:15.241191 26757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:15.241624 26749 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2528f951cfb4cc4ac5cd8ab0cec4b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } } }
I20260619 22:30:15.241819 26749 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:15.244310 26757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:15.250568 26757 catalog_manager.cc:1347] Generated new cluster ID: 5ca4a77a2510492a9597863e3fc408b3
I20260619 22:30:15.250655 26757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:15.280838 26757 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:15.281200 26757 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:15.298512 26757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2528f951cfb4cc4ac5cd8ab0cec4b17: Generated new TSK 0
I20260619 22:30:15.299284 26757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:15.305490 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26670
2026-06-19T22:30:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:15 dist-test-slave-bt27 krb5kdc[26769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:15 dist-test-slave-bt27 krb5kdc[26769](info): set up 2 sockets
Jun 19 22:30:15 dist-test-slave-bt27 krb5kdc[26769](info): commencing operation
krb5kdc: starting...
W20260619 22:30:16.342478 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:16 dist-test-slave-bt27 krb5kdc[26769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:16.464972 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:16.575763 26782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:16.576105 26782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:16.576181 26782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:16.580353 26782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:16.580463 26782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:16.580503 26782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:16.580552 26782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:16.580587 26782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:16.586099 26782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:16.587316 26782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:16.588364 26782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:16.597138 26788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:16.597137 26787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:16.597172 26790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:16.597275 26782 server_base.cc:1034] running on GCE node
I20260619 22:30:16.598026 26782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:16.599233 26782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:16.600458 26782 hybrid_clock.cc:648] HybridClock initialized: now 1781908216600440 us; error 28 us; skew 500 ppm
Jun 19 22:30:16 dist-test-slave-bt27 krb5kdc[26769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:16.603442 26782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:16.604457 26782 webserver.cc:460] Webserver started at http://127.0.0.1:44043/ using document root <none> and password file <none>
I20260619 22:30:16.605018 26782 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:16.605082 26782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:16.605255 26782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:16.607437 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "96496f54f22e4856a906ca82b85f85dc"
format_stamp: "Formatted at 2026-06-19 22:30:16 on dist-test-slave-bt27"
server_key: "c22b4fc702bf243d6f7efb0874e84662"
server_key_iv: "eb51952964fb410665b418ae0185f2ce"
server_key_version: "encryptionkey@0"
I20260619 22:30:16.607997 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "96496f54f22e4856a906ca82b85f85dc"
format_stamp: "Formatted at 2026-06-19 22:30:16 on dist-test-slave-bt27"
server_key: "c22b4fc702bf243d6f7efb0874e84662"
server_key_iv: "eb51952964fb410665b418ae0185f2ce"
server_key_version: "encryptionkey@0"
I20260619 22:30:16.611975 26782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:16.614456 26797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:16.615712 26782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:16.615870 26782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "96496f54f22e4856a906ca82b85f85dc"
format_stamp: "Formatted at 2026-06-19 22:30:16 on dist-test-slave-bt27"
server_key: "c22b4fc702bf243d6f7efb0874e84662"
server_key_iv: "eb51952964fb410665b418ae0185f2ce"
server_key_version: "encryptionkey@0"
I20260619 22:30:16.615974 26782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:16.639006 26782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:16.642234 26782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:16.642436 26782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:16.649582 26782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36939
I20260619 22:30:16.649602 26849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36939 every 8 connection(s)
I20260619 22:30:16.650656 26782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:16.651889 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26782
I20260619 22:30:16.652045 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:16.652333 5827 external_mini_cluster.cc:1428] Setting key e80165ed28950e174554d1225ec26c48
I20260619 22:30:16.653726 26850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:16.659240 26850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc: Bootstrap starting.
Jun 19 22:30:16 dist-test-slave-bt27 krb5kdc[26769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:30:16.661512 26850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:16.662461 26850 log.cc:824] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:16.664362 26850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc: No bootstrap required, opened a new log
I20260619 22:30:16.666059 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:16.653738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:57483 (local address 127.0.0.1:36939)
0619 22:30:16.654216 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:16.654232 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:16.654268 (+ 36us) server_negotiation.cc:407] Connection header received
0619 22:30:16.654915 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:16.654930 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:16.655257 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:16.655589 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:16.656390 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:16.657367 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:16.658243 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:16.658705 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:16.658879 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:16.661388 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:16.661393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:16.661406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:16.661434 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:16.663186 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:16.663698 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:16.663703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:16.663707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:16.663837 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:16.664170 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:16.664178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:16.664180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:16.664614 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:16.664833 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:16.665100 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 22:30:16.665323 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260619 22:30:16.668282 26850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96496f54f22e4856a906ca82b85f85dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260619 22:30:16.668550 26850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:16.668623 26850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96496f54f22e4856a906ca82b85f85dc, State: Initialized, Role: FOLLOWER
I20260619 22:30:16.668980 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:16.669132 26850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96496f54f22e4856a906ca82b85f85dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260619 22:30:16.669281 26850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:16.669346 26850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:16.669448 26850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:16.670483 26850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96496f54f22e4856a906ca82b85f85dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260619 22:30:16.670822 26850 leader_election.cc:302] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96496f54f22e4856a906ca82b85f85dc; no voters:
I20260619 22:30:16.671159 26850 leader_election.cc:288] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:16.671249 26859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:16.671478 26859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [term 1 LEADER]: Becoming Leader. State: Replica: 96496f54f22e4856a906ca82b85f85dc, State: Running, Role: LEADER
I20260619 22:30:16.671841 26859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96496f54f22e4856a906ca82b85f85dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260619 22:30:16.672585 26850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:16.672987 26859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96496f54f22e4856a906ca82b85f85dc. Latest consensus state: current_term: 1 leader_uuid: "96496f54f22e4856a906ca82b85f85dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96496f54f22e4856a906ca82b85f85dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } } }
I20260619 22:30:16.673156 26859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:16.673604 26864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:16.675634 26862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96496f54f22e4856a906ca82b85f85dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96496f54f22e4856a906ca82b85f85dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } } }
I20260619 22:30:16.675750 26862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:16.676831 26864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:16.680529 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:16.671119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:38595 (local address 127.0.0.1:36939)
0619 22:30:16.671233 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:16.671239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:16.671608 (+ 369us) server_negotiation.cc:407] Connection header received
0619 22:30:16.671791 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:16.671795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:16.671856 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:16.671944 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:16.672711 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:16.673513 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:16.674655 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:16.674996 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:16.675099 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:16.676857 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:16.676862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:16.676867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:16.676898 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:16.678572 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:16.679158 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:16.679162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:16.679163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:16.679250 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:16.679620 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:16.679623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:16.679624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:16.679839 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:16.679970 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:16.680280 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 22:30:16.680385 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 22:30:16.682955 26864 catalog_manager.cc:1347] Generated new cluster ID: 5ebc57bf84aa4145b7bbc06631c6aebc
I20260619 22:30:16.683019 26864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:16.695511 26864 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:16.696525 26864 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:16.705647 26864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96496f54f22e4856a906ca82b85f85dc: Generated new TSK 0
I20260619 22:30:16.706498 26864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:16.719838 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26782
W20260619 22:30:16.725592 26767 connection.cc:535] client connection to 127.0.0.1:36939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36939): Connection reset by peer (error 104)
2026-06-19T22:30:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:16 dist-test-slave-bt27 krb5kdc[26881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:16 dist-test-slave-bt27 krb5kdc[26881](info): set up 2 sockets
Jun 19 22:30:16 dist-test-slave-bt27 krb5kdc[26881](info): commencing operation
krb5kdc: starting...
W20260619 22:30:17.770558 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:17 dist-test-slave-bt27 krb5kdc[26881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:17.895489 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:18.004947 26894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:18.005298 26894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:18.005378 26894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:18.009490 26894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:18.009600 26894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:18.009639 26894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:18.009673 26894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:18.009706 26894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:18.015252 26894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:18.016394 26894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:18.017546 26894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:18.026495 26899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:18.026530 26902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:18.026784 26900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:18.026892 26894 server_base.cc:1034] running on GCE node
I20260619 22:30:18.027488 26894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:18.028650 26894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:18.029834 26894 hybrid_clock.cc:648] HybridClock initialized: now 1781908218029813 us; error 32 us; skew 500 ppm
Jun 19 22:30:18 dist-test-slave-bt27 krb5kdc[26881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:18.036186 26894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:18.037214 26894 webserver.cc:460] Webserver started at http://127.0.0.1:33189/ using document root <none> and password file <none>
I20260619 22:30:18.037803 26894 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:18.037865 26894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:18.038085 26894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:18.040273 26894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "d0b02faea8e4481baf68aab697c7de2c"
format_stamp: "Formatted at 2026-06-19 22:30:18 on dist-test-slave-bt27"
server_key: "e8d9b3f2e2b40409893c01687f9aa26b"
server_key_iv: "ea572b2b1bc5d453b07ee2134df6becf"
server_key_version: "encryptionkey@0"
I20260619 22:30:18.040853 26894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "d0b02faea8e4481baf68aab697c7de2c"
format_stamp: "Formatted at 2026-06-19 22:30:18 on dist-test-slave-bt27"
server_key: "e8d9b3f2e2b40409893c01687f9aa26b"
server_key_iv: "ea572b2b1bc5d453b07ee2134df6becf"
server_key_version: "encryptionkey@0"
I20260619 22:30:18.044895 26894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:18.047636 26909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:18.048993 26894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:30:18.049144 26894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "d0b02faea8e4481baf68aab697c7de2c"
format_stamp: "Formatted at 2026-06-19 22:30:18 on dist-test-slave-bt27"
server_key: "e8d9b3f2e2b40409893c01687f9aa26b"
server_key_iv: "ea572b2b1bc5d453b07ee2134df6becf"
server_key_version: "encryptionkey@0"
I20260619 22:30:18.049273 26894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:18.072949 26894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:18.076344 26894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:18.076578 26894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:18.084076 26894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42855
I20260619 22:30:18.084098 26961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42855 every 8 connection(s)
I20260619 22:30:18.085078 26894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:18.088053 26962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:18.092324 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 26894
I20260619 22:30:18.092494 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:18.092831 5827 external_mini_cluster.cc:1428] Setting key c2f399d8c89e2e23a3162b4255b08841
I20260619 22:30:18.094269 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c: Bootstrap starting.
I20260619 22:30:18.097198 26962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:18.098188 26962 log.cc:824] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:18.100126 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c: No bootstrap required, opened a new log
I20260619 22:30:18.103178 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:18.094542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:53069 (local address 127.0.0.1:42855)
0619 22:30:18.095033 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:18.095055 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:18.095099 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:30:18.096040 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:18.096064 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:18.096515 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:18.096887 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:18.097362 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:18.098836 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:18.099775 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:18.100298 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:18.100472 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:18.101456 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:18.101462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:18.101475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:18.101870 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:18.101930 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:18.101954 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:18.102106 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:18.102324 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 22:30:18.102468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260619 22:30:18.103924 26962 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b02faea8e4481baf68aab697c7de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42855 } }
I20260619 22:30:18.104189 26962 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:18.104239 26962 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b02faea8e4481baf68aab697c7de2c, State: Initialized, Role: FOLLOWER
I20260619 22:30:18.104707 26962 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b02faea8e4481baf68aab697c7de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42855 } }
I20260619 22:30:18.104852 26962 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:18.104939 26962 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:18.105044 26962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:18.106178 26962 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b02faea8e4481baf68aab697c7de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42855 } }
I20260619 22:30:18.106309 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:18.106498 26962 leader_election.cc:302] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b02faea8e4481baf68aab697c7de2c; no voters:
I20260619 22:30:18.106791 26962 leader_election.cc:288] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:18.106907 26967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:18.107115 26967 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [term 1 LEADER]: Becoming Leader. State: Replica: d0b02faea8e4481baf68aab697c7de2c, State: Running, Role: LEADER
I20260619 22:30:18.107472 26967 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b02faea8e4481baf68aab697c7de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42855 } }
I20260619 22:30:18.108013 26962 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:18.108741 26967 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b02faea8e4481baf68aab697c7de2c. Latest consensus state: current_term: 1 leader_uuid: "d0b02faea8e4481baf68aab697c7de2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b02faea8e4481baf68aab697c7de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42855 } } }
I20260619 22:30:18.109025 26967 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:18.109223 26968 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b02faea8e4481baf68aab697c7de2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b02faea8e4481baf68aab697c7de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42855 } } }
I20260619 22:30:18.109320 26968 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:18.109488 26981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:18.112886 26981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:18.114275 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:18.108553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:49251 (local address 127.0.0.1:42855)
0619 22:30:18.108666 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:18.108673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:18.109000 (+ 327us) server_negotiation.cc:407] Connection header received
0619 22:30:18.109208 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:18.109214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:18.109274 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:18.109333 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:18.109793 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:18.110700 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:18.112255 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:18.112582 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:18.112697 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:18.113665 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:18.113669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:18.113672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:18.113687 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:18.113717 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:18.113731 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:18.113837 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:18.114062 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 22:30:18.114101 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:30:18.119305 26981 catalog_manager.cc:1347] Generated new cluster ID: a040e06ce7d84083b42478133a88fe2f
I20260619 22:30:18.119388 26981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:18.141148 26981 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:18.142120 26981 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:18.146833 26981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0b02faea8e4481baf68aab697c7de2c: Generated new TSK 0
I20260619 22:30:18.147500 26981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:18.161584 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 26894
W20260619 22:30:18.167487 26879 connection.cc:535] client connection to 127.0.0.1:42855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42855): Connection reset by peer (error 104)
2026-06-19T22:30:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:18 dist-test-slave-bt27 krb5kdc[26993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:18 dist-test-slave-bt27 krb5kdc[26993](info): set up 2 sockets
Jun 19 22:30:18 dist-test-slave-bt27 krb5kdc[26993](info): commencing operation
krb5kdc: starting...
W20260619 22:30:19.210513 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:19 dist-test-slave-bt27 krb5kdc[26993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:19.337556 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:19.445967 27006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:19.446266 27006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:19.446311 27006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:19.450230 27006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:19.450286 27006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:19.450306 27006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:19.450323 27006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:19.450338 27006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:19.455622 27006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:19.456629 27006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:19.457612 27006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:19.466117 27011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:19.466284 27006 server_base.cc:1034] running on GCE node
W20260619 22:30:19.466179 27014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:19.466153 27012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:19.467185 27006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:19.468354 27006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:19.469564 27006 hybrid_clock.cc:648] HybridClock initialized: now 1781908219469532 us; error 44 us; skew 500 ppm
Jun 19 22:30:19 dist-test-slave-bt27 krb5kdc[26993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:19.472504 27006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:19.473410 27006 webserver.cc:460] Webserver started at http://127.0.0.1:34479/ using document root <none> and password file <none>
I20260619 22:30:19.473941 27006 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:19.474001 27006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:19.474224 27006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:19.476303 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "f9d946512104401ab0427213c351ee12"
format_stamp: "Formatted at 2026-06-19 22:30:19 on dist-test-slave-bt27"
server_key: "f6c689a566038d598d2def418952d2f9"
server_key_iv: "6246b0d2b66ee7f8692cd12513d9ab4d"
server_key_version: "encryptionkey@0"
I20260619 22:30:19.476868 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "f9d946512104401ab0427213c351ee12"
format_stamp: "Formatted at 2026-06-19 22:30:19 on dist-test-slave-bt27"
server_key: "f6c689a566038d598d2def418952d2f9"
server_key_iv: "6246b0d2b66ee7f8692cd12513d9ab4d"
server_key_version: "encryptionkey@0"
I20260619 22:30:19.480679 27006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 22:30:19.483310 27021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:19.484715 27006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 22:30:19.484824 27006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "f9d946512104401ab0427213c351ee12"
format_stamp: "Formatted at 2026-06-19 22:30:19 on dist-test-slave-bt27"
server_key: "f6c689a566038d598d2def418952d2f9"
server_key_iv: "6246b0d2b66ee7f8692cd12513d9ab4d"
server_key_version: "encryptionkey@0"
I20260619 22:30:19.484935 27006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:19.489434 27006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:19.492406 27006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:19.492551 27006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:19.499357 27006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39839
I20260619 22:30:19.499432 27073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39839 every 8 connection(s)
I20260619 22:30:19.500304 27006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:19.503168 27074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:19.503824 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27006
I20260619 22:30:19.503979 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:19.504285 5827 external_mini_cluster.cc:1428] Setting key dceca38f4c29a773a707c56ba378f8d3
I20260619 22:30:19.510217 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:19.505787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:50187 (local address 127.0.0.1:39839)
0619 22:30:19.506279 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:19.506299 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:19.506346 (+ 47us) server_negotiation.cc:407] Connection header received
0619 22:30:19.507186 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:19.507210 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:19.507578 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:19.507962 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:19.508762 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:19.508768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:19.508785 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:19.508796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:19.508864 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:19.508896 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:19.509050 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:19.509119 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 22:30:19.509279 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260619 22:30:19.510258 27074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12: Bootstrap starting.
I20260619 22:30:19.512959 27074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:19.513293 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:19.513934 27074 log.cc:824] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:19.516358 27074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12: No bootstrap required, opened a new log
I20260619 22:30:19.517375 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:19.515434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:48037 (local address 127.0.0.1:39839)
0619 22:30:19.515572 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:19.515576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:19.515790 (+ 214us) server_negotiation.cc:407] Connection header received
0619 22:30:19.515904 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:19.515906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:19.515940 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:19.516012 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:19.516892 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:19.516895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:19.516897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:19.516908 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:19.516935 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:19.516946 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:19.517016 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:19.517157 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 22:30:19.517226 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 22:30:19.520565 27074 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d946512104401ab0427213c351ee12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39839 } }
I20260619 22:30:19.520848 27074 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:19.520910 27074 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d946512104401ab0427213c351ee12, State: Initialized, Role: FOLLOWER
I20260619 22:30:19.521374 27074 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d946512104401ab0427213c351ee12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39839 } }
I20260619 22:30:19.521531 27074 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:19.521602 27074 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:19.521703 27074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:19.522784 27074 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d946512104401ab0427213c351ee12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39839 } }
I20260619 22:30:19.523177 27074 leader_election.cc:302] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9d946512104401ab0427213c351ee12; no voters:
I20260619 22:30:19.523559 27074 leader_election.cc:288] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:19.523682 27084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:19.523937 27084 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [term 1 LEADER]: Becoming Leader. State: Replica: f9d946512104401ab0427213c351ee12, State: Running, Role: LEADER
I20260619 22:30:19.524279 27084 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d946512104401ab0427213c351ee12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39839 } }
I20260619 22:30:19.524850 27074 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:19.525945 27085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9d946512104401ab0427213c351ee12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d946512104401ab0427213c351ee12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39839 } } }
I20260619 22:30:19.526078 27085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:19.526401 27086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9d946512104401ab0427213c351ee12. Latest consensus state: current_term: 1 leader_uuid: "f9d946512104401ab0427213c351ee12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d946512104401ab0427213c351ee12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39839 } } }
I20260619 22:30:19.526502 27086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:19.526599 27093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:19.529771 27093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:19.535840 27093 catalog_manager.cc:1347] Generated new cluster ID: 9cf61d38ae27447580a5a8f4525c6489
I20260619 22:30:19.535920 27093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:19.550654 27093 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:19.551044 27093 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:19.568866 27093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9d946512104401ab0427213c351ee12: Generated new TSK 0
I20260619 22:30:19.569650 27093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:19.574637 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27006
2026-06-19T22:30:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:19 dist-test-slave-bt27 krb5kdc[27105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:19 dist-test-slave-bt27 krb5kdc[27105](info): set up 2 sockets
Jun 19 22:30:19 dist-test-slave-bt27 krb5kdc[27105](info): commencing operation
krb5kdc: starting...
W20260619 22:30:20.622464 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:20 dist-test-slave-bt27 krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:20.743468 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:20.853294 27118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:20.853607 27118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:20.853678 27118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:20.857726 27118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:20.857828 27118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:20.857867 27118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:20.857919 27118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:20.857970 27118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:20.863405 27118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:20.864568 27118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:20.865586 27118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:20.874051 27123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:20.874089 27126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:20.874179 27118 server_base.cc:1034] running on GCE node
W20260619 22:30:20.874063 27124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:20.874977 27118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:20.876096 27118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:20.877274 27118 hybrid_clock.cc:648] HybridClock initialized: now 1781908220877259 us; error 27 us; skew 500 ppm
Jun 19 22:30:20 dist-test-slave-bt27 krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:20.880245 27118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:20.881239 27118 webserver.cc:460] Webserver started at http://127.0.0.1:40585/ using document root <none> and password file <none>
I20260619 22:30:20.881815 27118 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:20.881881 27118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:20.882125 27118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:20.884229 27118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "b33a37f5fdbd4700b5c82fad2e986d8b"
format_stamp: "Formatted at 2026-06-19 22:30:20 on dist-test-slave-bt27"
server_key: "1d6512a505b0135a68ffea384d67bca6"
server_key_iv: "39644209388ffd8ebb724bd5709ef51d"
server_key_version: "encryptionkey@0"
I20260619 22:30:20.884800 27118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "b33a37f5fdbd4700b5c82fad2e986d8b"
format_stamp: "Formatted at 2026-06-19 22:30:20 on dist-test-slave-bt27"
server_key: "1d6512a505b0135a68ffea384d67bca6"
server_key_iv: "39644209388ffd8ebb724bd5709ef51d"
server_key_version: "encryptionkey@0"
I20260619 22:30:20.888682 27118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 22:30:20.891268 27133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:20.892580 27118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:20.892756 27118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "b33a37f5fdbd4700b5c82fad2e986d8b"
format_stamp: "Formatted at 2026-06-19 22:30:20 on dist-test-slave-bt27"
server_key: "1d6512a505b0135a68ffea384d67bca6"
server_key_iv: "39644209388ffd8ebb724bd5709ef51d"
server_key_version: "encryptionkey@0"
I20260619 22:30:20.892884 27118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:20.906512 27118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:20.909791 27118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:20.910004 27118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:20.917585 27118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41915
I20260619 22:30:20.917600 27185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41915 every 8 connection(s)
I20260619 22:30:20.918653 27118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:20.919842 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27118
I20260619 22:30:20.919981 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:20.920289 5827 external_mini_cluster.cc:1428] Setting key 374f388f2f9a397042d5c012674d968c
I20260619 22:30:20.922050 27186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:20.929032 27186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b: Bootstrap starting.
Jun 19 22:30:20 dist-test-slave-bt27 krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 22:30:20.931852 27186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:20.932811 27186 log.cc:824] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:20.935190 27186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b: No bootstrap required, opened a new log
I20260619 22:30:20.936445 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:20.921825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:51955 (local address 127.0.0.1:41915)
0619 22:30:20.922276 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:20.922296 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:20.922339 (+ 43us) server_negotiation.cc:407] Connection header received
0619 22:30:20.923163 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:20.923184 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:20.923626 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:20.923993 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:20.924756 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:20.926315 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:20.927243 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:20.927966 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:20.928139 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:20.931151 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:20.931161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:20.931178 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:20.931213 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:20.933218 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:20.933929 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:20.933936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:20.933942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:20.934081 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:20.934517 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:20.934520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:20.934522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:20.934946 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:20.935157 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:20.935495 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 22:30:20.935718 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260619 22:30:20.939601 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:20.939429 27186 raft_consensus.cc:348] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260619 22:30:20.939718 27186 raft_consensus.cc:374] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:20.939810 27186 raft_consensus.cc:729] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33a37f5fdbd4700b5c82fad2e986d8b, State: Initialized, Role: FOLLOWER
I20260619 22:30:20.940218 27186 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260619 22:30:20.940369 27186 raft_consensus.cc:388] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:20.940445 27186 raft_consensus.cc:482] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:20.940544 27186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:20.941593 27186 raft_consensus.cc:504] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260619 22:30:20.942003 27186 leader_election.cc:302] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33a37f5fdbd4700b5c82fad2e986d8b; no voters:
I20260619 22:30:20.942440 27186 leader_election.cc:288] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:20.942531 27196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:20.942800 27196 raft_consensus.cc:686] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [term 1 LEADER]: Becoming Leader. State: Replica: b33a37f5fdbd4700b5c82fad2e986d8b, State: Running, Role: LEADER
I20260619 22:30:20.943166 27196 consensus_queue.cc:237] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260619 22:30:20.943854 27186 sys_catalog.cc:564] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:20.944402 27196 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260619 22:30:20.945124 27196 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:20.945008 27197 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b33a37f5fdbd4700b5c82fad2e986d8b. Latest consensus state: current_term: 1 leader_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a37f5fdbd4700b5c82fad2e986d8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260619 22:30:20.945442 27197 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b [sys.catalog]: This master's current role is: LEADER
W20260619 22:30:20.948841 27211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:20.948948 27211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:20.949008 27205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:20.949831 27205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:20.951013 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:20.941801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:34803 (local address 127.0.0.1:41915)
0619 22:30:20.941939 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:20.941944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:20.942115 (+ 171us) server_negotiation.cc:407] Connection header received
0619 22:30:20.942273 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:20.942275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:20.942319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:20.942386 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:20.943261 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:20.944082 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:20.945278 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:20.945686 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:20.945766 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:20.947628 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:20.947632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:20.947635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:20.947660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:20.949230 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:20.949696 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:20.949700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:20.949702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:20.949780 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:20.950102 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:20.950105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:20.950107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:20.950338 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:20.950486 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:20.950746 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 22:30:20.950853 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 22:30:20.955997 27205 catalog_manager.cc:1347] Generated new cluster ID: 98b4d7c378cd4b548ed9021936a9e258
I20260619 22:30:20.956091 27205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:20.966454 27205 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:20.967685 27205 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:20.976720 27205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b33a37f5fdbd4700b5c82fad2e986d8b: Generated new TSK 0
I20260619 22:30:20.977804 27205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:20.990788 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27118
W20260619 22:30:20.996356 27103 connection.cc:535] client connection to 127.0.0.1:41915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41915): Connection reset by peer (error 104)
2026-06-19T22:30:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:21 dist-test-slave-bt27 krb5kdc[27217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:21 dist-test-slave-bt27 krb5kdc[27217](info): set up 2 sockets
Jun 19 22:30:21 dist-test-slave-bt27 krb5kdc[27217](info): commencing operation
krb5kdc: starting...
W20260619 22:30:22.030471 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:22 dist-test-slave-bt27 krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:22.153333 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:22.265587 27230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:22.265931 27230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:22.266011 27230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:22.270169 27230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:22.270254 27230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:22.270292 27230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:22.270345 27230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:22.270381 27230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:22.275701 27230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:22.276811 27230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:22.277863 27230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:22.286443 27236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:22.286461 27235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:22.286489 27238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:22.286669 27230 server_base.cc:1034] running on GCE node
I20260619 22:30:22.287392 27230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:22.288582 27230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:22.289783 27230 hybrid_clock.cc:648] HybridClock initialized: now 1781908222289770 us; error 41 us; skew 500 ppm
Jun 19 22:30:22 dist-test-slave-bt27 krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:22.292748 27230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:22.293738 27230 webserver.cc:460] Webserver started at http://127.0.0.1:42787/ using document root <none> and password file <none>
I20260619 22:30:22.294368 27230 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:22.294464 27230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:22.294688 27230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:22.296808 27230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "52d84b70f4204335878854826feca5e6"
format_stamp: "Formatted at 2026-06-19 22:30:22 on dist-test-slave-bt27"
server_key: "83eaa71c12b330db37e8d6b6f1d40510"
server_key_iv: "39b4e17a799116f8762a167f80f64800"
server_key_version: "encryptionkey@0"
I20260619 22:30:22.297391 27230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "52d84b70f4204335878854826feca5e6"
format_stamp: "Formatted at 2026-06-19 22:30:22 on dist-test-slave-bt27"
server_key: "83eaa71c12b330db37e8d6b6f1d40510"
server_key_iv: "39b4e17a799116f8762a167f80f64800"
server_key_version: "encryptionkey@0"
I20260619 22:30:22.302078 27230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 22:30:22.304706 27245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:22.306051 27230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:30:22.306244 27230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "52d84b70f4204335878854826feca5e6"
format_stamp: "Formatted at 2026-06-19 22:30:22 on dist-test-slave-bt27"
server_key: "83eaa71c12b330db37e8d6b6f1d40510"
server_key_iv: "39b4e17a799116f8762a167f80f64800"
server_key_version: "encryptionkey@0"
I20260619 22:30:22.306376 27230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:22.326395 27230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:22.329797 27230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:22.330004 27230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:22.337446 27230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44799
I20260619 22:30:22.337467 27297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44799 every 8 connection(s)
I20260619 22:30:22.338537 27230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:22.340098 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27230
I20260619 22:30:22.340215 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:22.340528 5827 external_mini_cluster.cc:1428] Setting key a9c08d3638991af11dc2fc9cdbfe2f3a
I20260619 22:30:22.341764 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:22.348682 27298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6: Bootstrap starting.
I20260619 22:30:22.351040 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:22.342034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:57713 (local address 127.0.0.1:44799)
0619 22:30:22.342511 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:22.342531 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:22.342575 (+ 44us) server_negotiation.cc:407] Connection header received
0619 22:30:22.343454 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:22.343478 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:22.343913 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:22.344294 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:22.344720 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:22.346212 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:22.347113 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:22.347832 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:22.347994 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:22.348866 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:22.348871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:22.348887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:22.349388 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:22.349453 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:22.349484 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:22.349611 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:22.349787 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 22:30:22.349982 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":137,"threads_started":1}
I20260619 22:30:22.351553 27298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:22.352368 27298 log.cc:824] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:22.354166 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:22.354636 27298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6: No bootstrap required, opened a new log
I20260619 22:30:22.358963 27298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d84b70f4204335878854826feca5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260619 22:30:22.359268 27298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:22.359335 27298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d84b70f4204335878854826feca5e6, State: Initialized, Role: FOLLOWER
I20260619 22:30:22.359783 27298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d84b70f4204335878854826feca5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260619 22:30:22.359908 27298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:22.360031 27298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:22.360149 27298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:22.360795 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:22.356403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:41401 (local address 127.0.0.1:44799)
0619 22:30:22.356556 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:22.356562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:22.356712 (+ 150us) server_negotiation.cc:407] Connection header received
0619 22:30:22.356833 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:22.356838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:22.356901 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:22.357005 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:22.357292 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:22.358086 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:22.358958 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:22.359284 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:22.359397 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:22.360363 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:22.360369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:22.360375 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:22.360391 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:22.360422 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:22.360434 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:22.360527 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:22.360617 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 22:30:22.360654 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:30:22.361312 27298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d84b70f4204335878854826feca5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260619 22:30:22.361676 27298 leader_election.cc:302] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d84b70f4204335878854826feca5e6; no voters:
I20260619 22:30:22.362023 27298 leader_election.cc:288] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:22.362123 27308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:22.362378 27308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [term 1 LEADER]: Becoming Leader. State: Replica: 52d84b70f4204335878854826feca5e6, State: Running, Role: LEADER
I20260619 22:30:22.362711 27308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d84b70f4204335878854826feca5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260619 22:30:22.363438 27298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:22.364324 27309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d84b70f4204335878854826feca5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d84b70f4204335878854826feca5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } } }
I20260619 22:30:22.364470 27309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:22.364827 27310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d84b70f4204335878854826feca5e6. Latest consensus state: current_term: 1 leader_uuid: "52d84b70f4204335878854826feca5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d84b70f4204335878854826feca5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } } }
I20260619 22:30:22.364950 27310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:22.364864 27317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:22.368233 27317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:22.373353 27317 catalog_manager.cc:1347] Generated new cluster ID: 9c57612305e54a3186c1ac62a7bd0f98
I20260619 22:30:22.373422 27317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:22.387965 27317 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:22.388967 27317 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:22.400835 27317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d84b70f4204335878854826feca5e6: Generated new TSK 0
I20260619 22:30:22.401614 27317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:22.413719 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27230
W20260619 22:30:22.419292 27215 connection.cc:535] client connection to 127.0.0.1:44799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44799): Connection reset by peer (error 104)
2026-06-19T22:30:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:22 dist-test-slave-bt27 krb5kdc[27329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:22 dist-test-slave-bt27 krb5kdc[27329](info): set up 2 sockets
Jun 19 22:30:22 dist-test-slave-bt27 krb5kdc[27329](info): commencing operation
krb5kdc: starting...
W20260619 22:30:23.454352 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:23 dist-test-slave-bt27 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 22:30:23.573401 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:23.688529 27342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:23.688832 27342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:23.688876 27342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:23.692893 27342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:23.692958 27342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:23.692978 27342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:23.692996 27342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:23.693010 27342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:23.698401 27342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:23.699440 27342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:23.700403 27342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:23.709143 27347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:23.709143 27348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:23.709174 27350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:23.709362 27342 server_base.cc:1034] running on GCE node
I20260619 22:30:23.710222 27342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:23.711467 27342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:23.712699 27342 hybrid_clock.cc:648] HybridClock initialized: now 1781908223712660 us; error 35 us; skew 500 ppm
Jun 19 22:30:23 dist-test-slave-bt27 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:23.720275 27342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 22:30:23.721235 27342 webserver.cc:460] Webserver started at http://127.0.0.1:37997/ using document root <none> and password file <none>
I20260619 22:30:23.721822 27342 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:23.721886 27342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:23.722070 27342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:23.724249 27342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "c8b1d9d1f39846048b5987aaca2b4b7e"
format_stamp: "Formatted at 2026-06-19 22:30:23 on dist-test-slave-bt27"
server_key: "21b50f8381b58470d7ce6ccb0b69003a"
server_key_iv: "cae50ffc2aa0ab897e688cf201972fa9"
server_key_version: "encryptionkey@0"
I20260619 22:30:23.724779 27342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "c8b1d9d1f39846048b5987aaca2b4b7e"
format_stamp: "Formatted at 2026-06-19 22:30:23 on dist-test-slave-bt27"
server_key: "21b50f8381b58470d7ce6ccb0b69003a"
server_key_iv: "cae50ffc2aa0ab897e688cf201972fa9"
server_key_version: "encryptionkey@0"
I20260619 22:30:23.728507 27342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:23.730994 27357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:23.732219 27342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:23.732323 27342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "c8b1d9d1f39846048b5987aaca2b4b7e"
format_stamp: "Formatted at 2026-06-19 22:30:23 on dist-test-slave-bt27"
server_key: "21b50f8381b58470d7ce6ccb0b69003a"
server_key_iv: "cae50ffc2aa0ab897e688cf201972fa9"
server_key_version: "encryptionkey@0"
I20260619 22:30:23.732409 27342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:23.736843 27342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:23.739627 27342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:23.739776 27342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:23.746520 27342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38681
I20260619 22:30:23.746544 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38681 every 8 connection(s)
I20260619 22:30:23.747386 27342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:23.749936 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27342
I20260619 22:30:23.750079 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:23.750098 27410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:23.750422 5827 external_mini_cluster.cc:1428] Setting key 0b9f25a9ab9fae5afde446e121432a10
I20260619 22:30:23.755332 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:23.751758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:40077 (local address 127.0.0.1:38681)
0619 22:30:23.752194 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:23.752207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:23.752241 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:30:23.752865 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:23.752881 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:23.753141 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:23.753456 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:23.754226 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:23.754231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:23.754248 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:23.754258 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:23.754307 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:23.754325 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:23.754462 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:23.754529 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 22:30:23.754680 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260619 22:30:23.756734 27410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e: Bootstrap starting.
I20260619 22:30:23.757980 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:23.759093 27410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:23.759902 27410 log.cc:824] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:23.761868 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:23.760129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:36509 (local address 127.0.0.1:38681)
0619 22:30:23.760260 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:23.760266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:23.760469 (+ 203us) server_negotiation.cc:407] Connection header received
0619 22:30:23.760548 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:23.760555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:23.760598 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:23.760708 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:23.761391 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:23.761394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:23.761396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:23.761407 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:23.761433 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:23.761446 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:23.761536 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:23.761629 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 22:30:23.761664 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 22:30:23.762169 27410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e: No bootstrap required, opened a new log
I20260619 22:30:23.765659 27410 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38681 } }
I20260619 22:30:23.765908 27410 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:23.765980 27410 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b1d9d1f39846048b5987aaca2b4b7e, State: Initialized, Role: FOLLOWER
I20260619 22:30:23.766494 27410 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38681 } }
I20260619 22:30:23.766642 27410 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:23.766708 27410 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:23.766805 27410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:23.767791 27410 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38681 } }
I20260619 22:30:23.768134 27410 leader_election.cc:302] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8b1d9d1f39846048b5987aaca2b4b7e; no voters:
I20260619 22:30:23.768455 27410 leader_election.cc:288] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:23.768581 27420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:23.768814 27420 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [term 1 LEADER]: Becoming Leader. State: Replica: c8b1d9d1f39846048b5987aaca2b4b7e, State: Running, Role: LEADER
I20260619 22:30:23.769114 27420 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38681 } }
I20260619 22:30:23.769675 27410 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:23.770882 27422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8b1d9d1f39846048b5987aaca2b4b7e. Latest consensus state: current_term: 1 leader_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38681 } } }
I20260619 22:30:23.770958 27421 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b1d9d1f39846048b5987aaca2b4b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38681 } } }
I20260619 22:30:23.771015 27422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:23.771023 27421 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:23.771427 27429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:23.774348 27429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:23.780217 27429 catalog_manager.cc:1347] Generated new cluster ID: eddd5356fed941ddb4ffcc2c5dbd1bd3
I20260619 22:30:23.780303 27429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:23.790545 27429 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:23.790915 27429 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:23.796828 27429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8b1d9d1f39846048b5987aaca2b4b7e: Generated new TSK 0
I20260619 22:30:23.797502 27429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:23.806584 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27342
2026-06-19T22:30:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:23 dist-test-slave-bt27 krb5kdc[27441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:23 dist-test-slave-bt27 krb5kdc[27441](info): set up 2 sockets
Jun 19 22:30:23 dist-test-slave-bt27 krb5kdc[27441](info): commencing operation
krb5kdc: starting...
W20260619 22:30:24.850419 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:24 dist-test-slave-bt27 krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:24.978228 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38369
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:25.092345 27454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:25.092682 27454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:25.092757 27454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:25.096837 27454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:25.096946 27454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:25.096993 27454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:25.097031 27454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:25.097066 27454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:25.102454 27454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:38369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:25.103634 27454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:25.104753 27454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:25.113708 27462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:25.113708 27460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:25.113790 27459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:25.113835 27454 server_base.cc:1034] running on GCE node
I20260619 22:30:25.114686 27454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:25.115868 27454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:25.117077 27454 hybrid_clock.cc:648] HybridClock initialized: now 1781908225117064 us; error 28 us; skew 500 ppm
Jun 19 22:30:25 dist-test-slave-bt27 krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:25.120113 27454 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:25.121102 27454 webserver.cc:460] Webserver started at http://127.5.176.254:39885/ using document root <none> and password file <none>
I20260619 22:30:25.121692 27454 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:25.121788 27454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:25.122030 27454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:25.124281 27454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "0c3c8db033434ce097942809cbe88a5c"
format_stamp: "Formatted at 2026-06-19 22:30:25 on dist-test-slave-bt27"
server_key: "79c92f8884853932dc98167e60249875"
server_key_iv: "2a8f23bf28d440509ed9f88e5bdcb7bd"
server_key_version: "encryptionkey@0"
I20260619 22:30:25.124898 27454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "0c3c8db033434ce097942809cbe88a5c"
format_stamp: "Formatted at 2026-06-19 22:30:25 on dist-test-slave-bt27"
server_key: "79c92f8884853932dc98167e60249875"
server_key_iv: "2a8f23bf28d440509ed9f88e5bdcb7bd"
server_key_version: "encryptionkey@0"
I20260619 22:30:25.129010 27454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:30:25.131852 27469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:25.133249 27454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 22:30:25.133445 27454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "0c3c8db033434ce097942809cbe88a5c"
format_stamp: "Formatted at 2026-06-19 22:30:25 on dist-test-slave-bt27"
server_key: "79c92f8884853932dc98167e60249875"
server_key_iv: "2a8f23bf28d440509ed9f88e5bdcb7bd"
server_key_version: "encryptionkey@0"
I20260619 22:30:25.133592 27454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:25.171790 27454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:25.175661 27454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:25.175907 27454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:25.184000 27454 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:38369
I20260619 22:30:25.184084 27521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:38369 every 8 connection(s)
I20260619 22:30:25.185191 27454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:25.188179 27522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:25.194366 27522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c: Bootstrap starting.
I20260619 22:30:25.195446 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27454
I20260619 22:30:25.195616 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:25.195907 5827 external_mini_cluster.cc:1428] Setting key 53e305a2aeaf1318f6b23c544a0eb25f
I20260619 22:30:25.197376 27522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:25.198205 27522 log.cc:824] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:25.200456 27522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c: No bootstrap required, opened a new log
I20260619 22:30:25.205101 27522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c8db033434ce097942809cbe88a5c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38369 } }
Jun 19 22:30:25 dist-test-slave-bt27 krb5kdc[27441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:25.205358 27522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:25.205451 27522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3c8db033434ce097942809cbe88a5c, State: Initialized, Role: FOLLOWER
I20260619 22:30:25.205981 27522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c8db033434ce097942809cbe88a5c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38369 } }
I20260619 22:30:25.206172 27522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:25.206245 27522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:25.206349 27522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:25.207281 27522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c8db033434ce097942809cbe88a5c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38369 } }
I20260619 22:30:25.207648 27522 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3c8db033434ce097942809cbe88a5c; no voters:
I20260619 22:30:25.208021 27522 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:25.208207 27527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:25.208436 27527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [term 1 LEADER]: Becoming Leader. State: Replica: 0c3c8db033434ce097942809cbe88a5c, State: Running, Role: LEADER
I20260619 22:30:25.208782 27527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c8db033434ce097942809cbe88a5c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38369 } }
I20260619 22:30:25.209445 27522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:25.210160 27527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3c8db033434ce097942809cbe88a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c8db033434ce097942809cbe88a5c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38369 } } }
I20260619 22:30:25.210314 27527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:25.210546 27527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3c8db033434ce097942809cbe88a5c. Latest consensus state: current_term: 1 leader_uuid: "0c3c8db033434ce097942809cbe88a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c8db033434ce097942809cbe88a5c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 38369 } } }
I20260619 22:30:25.210655 27527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:25.210960 27535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:25.212258 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:25.197137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48283 (local address 127.5.176.254:38369)
0619 22:30:25.197612 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:25.197629 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:25.197663 (+ 34us) server_negotiation.cc:407] Connection header received
0619 22:30:25.198491 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:25.198511 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:25.198850 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:25.199134 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:25.200037 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:25.201117 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:25.202332 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:25.202834 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:25.202975 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:25.206071 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:25.206076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:25.206089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:25.206154 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:25.208237 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:25.208931 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:25.208937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:25.208943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:25.209097 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:25.209492 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:25.209495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:25.209496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:25.210026 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:25.210908 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:25.211086 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 22:30:25.211308 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":160,"threads_started":1}
I20260619 22:30:25.215235 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:25.215337 27535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:25.221704 27535 catalog_manager.cc:1347] Generated new cluster ID: f6c03ae1294c4eafac9efe5160e4fa8f
I20260619 22:30:25.221776 27535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:25.228111 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:25.217275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53889 (local address 127.5.176.254:38369)
0619 22:30:25.217386 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:25.217391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:25.219749 (+ 2358us) server_negotiation.cc:407] Connection header received
0619 22:30:25.219889 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:25.219895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:25.219959 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:25.220065 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:25.220734 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:25.221270 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:25.222335 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:25.222711 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:25.222825 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:25.224485 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:25.224489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:25.224492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:25.224522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:25.226264 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:25.226806 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:25.226810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:25.226812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:25.226901 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:25.227214 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:25.227217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:25.227219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:25.227436 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:25.227597 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:25.227841 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 22:30:25.227946 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 22:30:25.233808 27535 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:25.234834 27535 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:25.247444 27535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3c8db033434ce097942809cbe88a5c: Generated new TSK 0
I20260619 22:30:25.248385 27535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:25.259090 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27454
W20260619 22:30:25.266405 27439 connection.cc:535] client connection to 127.5.176.254:38369 recv error: Network error: failed to read from TLS socket (remote: 127.5.176.254:38369): Connection reset by peer (error 104)
2026-06-19T22:30:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:25 dist-test-slave-bt27 krb5kdc[27553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:25 dist-test-slave-bt27 krb5kdc[27553](info): set up 2 sockets
Jun 19 22:30:25 dist-test-slave-bt27 krb5kdc[27553](info): commencing operation
krb5kdc: starting...
W20260619 22:30:26.310494 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:26 dist-test-slave-bt27 krb5kdc[27553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:26.437183 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39301
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:26.549404 27566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:26.549751 27566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:26.549827 27566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:26.554105 27566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:26.554231 27566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:26.554271 27566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:26.554330 27566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:26.554369 27566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:26.559906 27566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:39301
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:26.561077 27566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:26.562129 27566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:26.571281 27572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:26.571386 27566 server_base.cc:1034] running on GCE node
W20260619 22:30:26.571285 27574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:26.571309 27571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:26.572109 27566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:26.573243 27566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:26.574432 27566 hybrid_clock.cc:648] HybridClock initialized: now 1781908226574400 us; error 44 us; skew 500 ppm
Jun 19 22:30:26 dist-test-slave-bt27 krb5kdc[27553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908226, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:26.577220 27566 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:26.578125 27566 webserver.cc:460] Webserver started at http://127.5.176.254:34819/ using document root <none> and password file <none>
I20260619 22:30:26.578747 27566 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:26.578866 27566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:26.579102 27566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:26.581269 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "a44f324ff10a4fc2a19967b234dc7006"
format_stamp: "Formatted at 2026-06-19 22:30:26 on dist-test-slave-bt27"
server_key: "4e7e4cbf935566442d640838876fed4c"
server_key_iv: "9fafab2beae466b54f3883f6d3e4f178"
server_key_version: "encryptionkey@0"
I20260619 22:30:26.581849 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "a44f324ff10a4fc2a19967b234dc7006"
format_stamp: "Formatted at 2026-06-19 22:30:26 on dist-test-slave-bt27"
server_key: "4e7e4cbf935566442d640838876fed4c"
server_key_iv: "9fafab2beae466b54f3883f6d3e4f178"
server_key_version: "encryptionkey@0"
I20260619 22:30:26.585642 27566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 22:30:26.588197 27581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:26.589442 27566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:30:26.589604 27566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "a44f324ff10a4fc2a19967b234dc7006"
format_stamp: "Formatted at 2026-06-19 22:30:26 on dist-test-slave-bt27"
server_key: "4e7e4cbf935566442d640838876fed4c"
server_key_iv: "9fafab2beae466b54f3883f6d3e4f178"
server_key_version: "encryptionkey@0"
I20260619 22:30:26.589766 27566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:26.615317 27566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:26.619153 27566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:26.619397 27566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:26.627199 27566 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:39301
I20260619 22:30:26.627230 27633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:39301 every 8 connection(s)
I20260619 22:30:26.628214 27566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:26.631142 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:26.634279 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27566
I20260619 22:30:26.634436 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:26.634698 5827 external_mini_cluster.cc:1428] Setting key 64546695b97f4c6e074e2212ad45c766
I20260619 22:30:26.637418 27634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006: Bootstrap starting.
I20260619 22:30:26.639744 27634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:26.640496 27634 log.cc:824] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:26.642441 27634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006: No bootstrap required, opened a new log
I20260619 22:30:26.643921 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:26.636220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33393 (local address 127.5.176.254:39301)
0619 22:30:26.636672 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:26.636688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:26.636724 (+ 36us) server_negotiation.cc:407] Connection header received
0619 22:30:26.637366 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:26.637382 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:26.637839 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:26.638196 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:26.638625 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:26.639625 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:26.640583 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:26.641094 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:26.641285 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:26.642252 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:26.642261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:26.642280 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:26.642711 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:26.642764 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:26.642789 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:26.642927 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:26.643100 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 22:30:26.643237 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260619 22:30:26.646588 27634 raft_consensus.cc:348] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f324ff10a4fc2a19967b234dc7006" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39301 } }
I20260619 22:30:26.646867 27634 raft_consensus.cc:374] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:26.646944 27634 raft_consensus.cc:729] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f324ff10a4fc2a19967b234dc7006, State: Initialized, Role: FOLLOWER
I20260619 22:30:26.646991 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:26.647317 27634 consensus_queue.cc:260] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f324ff10a4fc2a19967b234dc7006" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39301 } }
I20260619 22:30:26.647424 27634 raft_consensus.cc:388] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:26.647461 27634 raft_consensus.cc:482] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:26.647554 27634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:26.648631 27634 raft_consensus.cc:504] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f324ff10a4fc2a19967b234dc7006" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39301 } }
I20260619 22:30:26.648969 27634 leader_election.cc:302] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a44f324ff10a4fc2a19967b234dc7006; no voters:
I20260619 22:30:26.649326 27634 leader_election.cc:288] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:26.649394 27644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:26.649632 27644 raft_consensus.cc:686] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [term 1 LEADER]: Becoming Leader. State: Replica: a44f324ff10a4fc2a19967b234dc7006, State: Running, Role: LEADER
I20260619 22:30:26.649963 27644 consensus_queue.cc:237] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f324ff10a4fc2a19967b234dc7006" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39301 } }
I20260619 22:30:26.650763 27634 sys_catalog.cc:564] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:26.651129 27646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a44f324ff10a4fc2a19967b234dc7006. Latest consensus state: current_term: 1 leader_uuid: "a44f324ff10a4fc2a19967b234dc7006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f324ff10a4fc2a19967b234dc7006" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39301 } } }
I20260619 22:30:26.651253 27646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:26.651592 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a44f324ff10a4fc2a19967b234dc7006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f324ff10a4fc2a19967b234dc7006" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39301 } } }
I20260619 22:30:26.651681 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:26.654310 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:26.649136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44763 (local address 127.5.176.254:39301)
0619 22:30:26.649253 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:26.649257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:26.649447 (+ 190us) server_negotiation.cc:407] Connection header received
0619 22:30:26.649653 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:26.649656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:26.649706 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:26.649756 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:26.650228 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:26.650966 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:26.652193 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:26.652521 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:26.652624 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:26.653581 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:26.653584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:26.653587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:26.653602 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:26.653630 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:26.653643 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:26.653735 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:26.654131 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 22:30:26.654192 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 22:30:26.656275 27659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:26.656344 27659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:26.656424 27648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:26.657095 27648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:26.662817 27648 catalog_manager.cc:1347] Generated new cluster ID: e1196a01f59f4ca294b0fa666bbad31e
I20260619 22:30:26.662922 27648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:26.688244 27648 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:26.689291 27648 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:26.701257 27648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a44f324ff10a4fc2a19967b234dc7006: Generated new TSK 0
I20260619 22:30:26.702239 27648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:26.707990 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27566
W20260619 22:30:26.713440 27551 connection.cc:535] client connection to 127.5.176.254:39301 recv error: Network error: failed to read from TLS socket (remote: 127.5.176.254:39301): Connection reset by peer (error 104)
2026-06-19T22:30:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:26 dist-test-slave-bt27 krb5kdc[27665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:26 dist-test-slave-bt27 krb5kdc[27665](info): set up 2 sockets
Jun 19 22:30:26 dist-test-slave-bt27 krb5kdc[27665](info): commencing operation
krb5kdc: starting...
W20260619 22:30:27.770424 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:27 dist-test-slave-bt27 krb5kdc[27665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:27.897034 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34435
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:28.008104 27678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:28.008453 27678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:28.008531 27678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:28.012718 27678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:28.012821 27678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:28.012857 27678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:28.012916 27678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:28.012957 27678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:28.018819 27678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:34435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:28.020000 27678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:28.021070 27678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:28.029810 27684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:28.029811 27683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:28.029882 27686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:28.030287 27678 server_base.cc:1034] running on GCE node
I20260619 22:30:28.030802 27678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:28.031934 27678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:28.033119 27678 hybrid_clock.cc:648] HybridClock initialized: now 1781908228033087 us; error 44 us; skew 500 ppm
Jun 19 22:30:28 dist-test-slave-bt27 krb5kdc[27665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:28.036047 27678 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:28.037034 27678 webserver.cc:460] Webserver started at http://127.5.176.254:35171/ using document root <none> and password file <none>
I20260619 22:30:28.037631 27678 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:28.037744 27678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:28.038002 27678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:28.040184 27678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "8f21a7e9bdc14210aa802f35f5778e5e"
format_stamp: "Formatted at 2026-06-19 22:30:28 on dist-test-slave-bt27"
server_key: "c0316411be7f9bfb4ae1079121f68615"
server_key_iv: "be0b980ea6fecce027dce01716c0507b"
server_key_version: "encryptionkey@0"
I20260619 22:30:28.040787 27678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "8f21a7e9bdc14210aa802f35f5778e5e"
format_stamp: "Formatted at 2026-06-19 22:30:28 on dist-test-slave-bt27"
server_key: "c0316411be7f9bfb4ae1079121f68615"
server_key_iv: "be0b980ea6fecce027dce01716c0507b"
server_key_version: "encryptionkey@0"
I20260619 22:30:28.044783 27678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 22:30:28.047257 27693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:28.048544 27678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 22:30:28.048712 27678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "8f21a7e9bdc14210aa802f35f5778e5e"
format_stamp: "Formatted at 2026-06-19 22:30:28 on dist-test-slave-bt27"
server_key: "c0316411be7f9bfb4ae1079121f68615"
server_key_iv: "be0b980ea6fecce027dce01716c0507b"
server_key_version: "encryptionkey@0"
I20260619 22:30:28.048844 27678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:28.053843 27678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:28.056878 27678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:28.057086 27678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:28.064455 27678 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:34435
I20260619 22:30:28.064473 27745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:34435 every 8 connection(s)
I20260619 22:30:28.065410 27678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:28.068202 27746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:28.073627 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27678
I20260619 22:30:28.073787 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:28.073885 27746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e: Bootstrap starting.
I20260619 22:30:28.074080 5827 external_mini_cluster.cc:1428] Setting key ea1b4e3b9455b1d160cb2dbb0bdcac3f
I20260619 22:30:28.076498 27746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:28.077224 27746 log.cc:824] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:28.079272 27746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e: No bootstrap required, opened a new log
I20260619 22:30:28.079391 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:28.075518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56003 (local address 127.5.176.254:34435)
0619 22:30:28.075988 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:28.076003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:28.076038 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:28.076746 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:28.076764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:28.077051 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:28.077386 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:28.078223 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:28.078227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:28.078238 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:28.078248 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:28.078302 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:28.078321 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:28.078441 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:28.078540 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 22:30:28.078698 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260619 22:30:28.082309 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:28.083441 27746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34435 } }
I20260619 22:30:28.083762 27746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:28.083841 27746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f21a7e9bdc14210aa802f35f5778e5e, State: Initialized, Role: FOLLOWER
I20260619 22:30:28.084326 27746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34435 } }
I20260619 22:30:28.084568 27746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:28.084666 27746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:28.084774 27746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:28.085776 27746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34435 } }
I20260619 22:30:28.086186 27746 leader_election.cc:302] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f21a7e9bdc14210aa802f35f5778e5e; no voters:
I20260619 22:30:28.086333 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:28.084438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43747 (local address 127.5.176.254:34435)
0619 22:30:28.084577 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:28.084583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:28.084828 (+ 245us) server_negotiation.cc:407] Connection header received
0619 22:30:28.084997 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:28.085000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:28.085044 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:28.085123 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:28.085896 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:28.085899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:28.085903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:28.085916 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:28.085945 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:28.085958 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:28.086036 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:28.086126 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 22:30:28.086198 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:30:28.086594 27746 leader_election.cc:288] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:28.086669 27756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:28.086887 27756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [term 1 LEADER]: Becoming Leader. State: Replica: 8f21a7e9bdc14210aa802f35f5778e5e, State: Running, Role: LEADER
I20260619 22:30:28.087191 27756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34435 } }
I20260619 22:30:28.087934 27746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:28.088868 27758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f21a7e9bdc14210aa802f35f5778e5e. Latest consensus state: current_term: 1 leader_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34435 } } }
I20260619 22:30:28.088905 27757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f21a7e9bdc14210aa802f35f5778e5e" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34435 } } }
I20260619 22:30:28.089076 27757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:28.089076 27758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:28.089661 27765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:28.092458 27765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:28.098358 27765 catalog_manager.cc:1347] Generated new cluster ID: ab9bec110f7f4a97925cfa0b42af8056
I20260619 22:30:28.098457 27765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:28.112350 27765 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:28.112704 27765 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:28.121567 27765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f21a7e9bdc14210aa802f35f5778e5e: Generated new TSK 0
I20260619 22:30:28.122388 27765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:28.130744 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27678
2026-06-19T22:30:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:28 dist-test-slave-bt27 krb5kdc[27777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:28 dist-test-slave-bt27 krb5kdc[27777](info): set up 2 sockets
Jun 19 22:30:28 dist-test-slave-bt27 krb5kdc[27777](info): commencing operation
krb5kdc: starting...
W20260619 22:30:29.218492 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:29 dist-test-slave-bt27 krb5kdc[27777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:29.345968 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39607
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:39607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:29.457723 27790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:29.457990 27790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:29.458065 27790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:29.462085 27790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:29.462162 27790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:29.462183 27790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:29.462253 27790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:29.462286 27790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:29.467732 27790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:39607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:29.468902 27790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:29.470031 27790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:29.478943 27798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:29.478950 27795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:29.478924 27796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:29.479298 27790 server_base.cc:1034] running on GCE node
I20260619 22:30:29.479938 27790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:29.481108 27790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:29.482306 27790 hybrid_clock.cc:648] HybridClock initialized: now 1781908229482283 us; error 34 us; skew 500 ppm
Jun 19 22:30:29 dist-test-slave-bt27 krb5kdc[27777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908229, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:29.488026 27790 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:29.488984 27790 webserver.cc:460] Webserver started at http://127.5.176.254:34341/ using document root <none> and password file <none>
I20260619 22:30:29.489526 27790 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:29.489586 27790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:29.489754 27790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:29.491834 27790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c"
format_stamp: "Formatted at 2026-06-19 22:30:29 on dist-test-slave-bt27"
server_key: "7db766080986d6b062eda44eb4c35cf0"
server_key_iv: "47a6d4a99d33a68ffbec740b10240889"
server_key_version: "encryptionkey@0"
I20260619 22:30:29.492372 27790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c"
format_stamp: "Formatted at 2026-06-19 22:30:29 on dist-test-slave-bt27"
server_key: "7db766080986d6b062eda44eb4c35cf0"
server_key_iv: "47a6d4a99d33a68ffbec740b10240889"
server_key_version: "encryptionkey@0"
I20260619 22:30:29.496261 27790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 22:30:29.498783 27805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:29.500144 27790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:29.500260 27790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c"
format_stamp: "Formatted at 2026-06-19 22:30:29 on dist-test-slave-bt27"
server_key: "7db766080986d6b062eda44eb4c35cf0"
server_key_iv: "47a6d4a99d33a68ffbec740b10240889"
server_key_version: "encryptionkey@0"
I20260619 22:30:29.500356 27790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:29.518612 27790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:29.521855 27790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:29.522025 27790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:29.529299 27790 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:39607
I20260619 22:30:29.529317 27857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:39607 every 8 connection(s)
I20260619 22:30:29.530315 27790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:29.532822 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27790
I20260619 22:30:29.532970 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:29.533282 5827 external_mini_cluster.cc:1428] Setting key 579d4c2223acfc9a48c78e649ee976da
I20260619 22:30:29.533367 27858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:29.540808 27858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c: Bootstrap starting.
Jun 19 22:30:29 dist-test-slave-bt27 krb5kdc[27777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:29.543344 27858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:29.544296 27858 log.cc:824] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:29.546900 27858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c: No bootstrap required, opened a new log
I20260619 22:30:29.549069 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:29.534760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60611 (local address 127.5.176.254:39607)
0619 22:30:29.535487 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:29.535506 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:29.535548 (+ 42us) server_negotiation.cc:407] Connection header received
0619 22:30:29.536242 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:29.536265 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:29.536698 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:29.537068 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:29.537729 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:29.539190 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:29.540024 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:29.540694 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:29.540856 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:29.543699 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:29.543706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:29.543724 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:29.543757 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:29.545760 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:29.546292 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:29.546298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:29.546305 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:29.546445 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:29.546779 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:29.546782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:29.546784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:29.547319 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:29.547506 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:29.547785 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 22:30:29.548094 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":130,"threads_started":1}
I20260619 22:30:29.551426 27858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39607 } }
I20260619 22:30:29.551716 27858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:29.551817 27858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431bbfb3a7874ca6bfcc5b281b02ec0c, State: Initialized, Role: FOLLOWER
I20260619 22:30:29.552237 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:29.552309 27858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39607 } }
I20260619 22:30:29.552502 27858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:29.552569 27858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:29.552666 27858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:29.553694 27858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39607 } }
I20260619 22:30:29.554044 27858 leader_election.cc:302] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431bbfb3a7874ca6bfcc5b281b02ec0c; no voters:
I20260619 22:30:29.554407 27858 leader_election.cc:288] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:29.554504 27867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:29.554785 27867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [term 1 LEADER]: Becoming Leader. State: Replica: 431bbfb3a7874ca6bfcc5b281b02ec0c, State: Running, Role: LEADER
I20260619 22:30:29.555154 27867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39607 } }
I20260619 22:30:29.555943 27858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:29.558722 27870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431bbfb3a7874ca6bfcc5b281b02ec0c. Latest consensus state: current_term: 1 leader_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39607 } } }
I20260619 22:30:29.559237 27870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:29.559113 27869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431bbfb3a7874ca6bfcc5b281b02ec0c" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39607 } } }
I20260619 22:30:29.559453 27869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c [sys.catalog]: This master's current role is: LEADER
W20260619 22:30:29.561625 27883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:29.561739 27883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:29.561831 27884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:29.562713 27884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:29.563356 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:29.554444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55321 (local address 127.5.176.254:39607)
0619 22:30:29.554756 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:29.554763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:29.554828 (+ 65us) server_negotiation.cc:407] Connection header received
0619 22:30:29.555005 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:29.555008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:29.555053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:29.555100 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:29.555974 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:29.556689 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:29.557892 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:29.558263 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:29.558360 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:29.560098 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:29.560102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:29.560104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:29.560128 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:29.561587 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:29.562082 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:29.562085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:29.562087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:29.562217 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:29.562505 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:29.562509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:29.562511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:29.562690 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:29.562825 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:29.563057 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 22:30:29.563175 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260619 22:30:29.568864 27884 catalog_manager.cc:1347] Generated new cluster ID: e5afcc41a5b949b9996df924202fdaba
I20260619 22:30:29.568953 27884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:29.603538 27884 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:29.604821 27884 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:29.610870 27884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431bbfb3a7874ca6bfcc5b281b02ec0c: Generated new TSK 0
I20260619 22:30:29.611586 27884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:29.623987 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27790
2026-06-19T22:30:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:29 dist-test-slave-bt27 krb5kdc[27889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:29 dist-test-slave-bt27 krb5kdc[27889](info): set up 2 sockets
Jun 19 22:30:29 dist-test-slave-bt27 krb5kdc[27889](info): commencing operation
krb5kdc: starting...
W20260619 22:30:30.695165 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:30 dist-test-slave-bt27 krb5kdc[27889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:30.821897 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39019
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:30.932070 27902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:30.932422 27902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:30.932500 27902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:30.936587 27902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:30.936697 27902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:30.936753 27902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:30.936806 27902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:30.936844 27902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:30.942233 27902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:30.943321 27902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:30.944345 27902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:30.952883 27908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:30.953058 27902 server_base.cc:1034] running on GCE node
W20260619 22:30:30.952950 27907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:30.953254 27910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:30.953799 27902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:30.954949 27902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:30.956151 27902 hybrid_clock.cc:648] HybridClock initialized: now 1781908230956107 us; error 55 us; skew 500 ppm
Jun 19 22:30:30 dist-test-slave-bt27 krb5kdc[27889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908230, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:30.958982 27902 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:30.959928 27902 webserver.cc:460] Webserver started at http://127.5.176.254:33329/ using document root <none> and password file <none>
I20260619 22:30:30.960510 27902 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:30.960604 27902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:30.960829 27902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:30.962947 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "b1090d78b64e4ff7a700addd62757172"
format_stamp: "Formatted at 2026-06-19 22:30:30 on dist-test-slave-bt27"
server_key: "9a0f7b111e9646a18b3177c059c18e20"
server_key_iv: "d9d30eb95add227095ea787f2221de8f"
server_key_version: "encryptionkey@0"
I20260619 22:30:30.963521 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "b1090d78b64e4ff7a700addd62757172"
format_stamp: "Formatted at 2026-06-19 22:30:30 on dist-test-slave-bt27"
server_key: "9a0f7b111e9646a18b3177c059c18e20"
server_key_iv: "d9d30eb95add227095ea787f2221de8f"
server_key_version: "encryptionkey@0"
I20260619 22:30:30.967264 27902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 22:30:30.969779 27917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:30.971071 27902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:30.971213 27902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "b1090d78b64e4ff7a700addd62757172"
format_stamp: "Formatted at 2026-06-19 22:30:30 on dist-test-slave-bt27"
server_key: "9a0f7b111e9646a18b3177c059c18e20"
server_key_iv: "d9d30eb95add227095ea787f2221de8f"
server_key_version: "encryptionkey@0"
I20260619 22:30:30.971334 27902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:30.986505 27902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:30.989475 27902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:30.989691 27902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:30.996599 27902 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:39019
I20260619 22:30:30.996646 27969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:39019 every 8 connection(s)
I20260619 22:30:30.997563 27902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:30.998423 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 27902
I20260619 22:30:30.998555 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:30.998852 5827 external_mini_cluster.cc:1428] Setting key b025513b34bc6c8ba11b5dea73eba40a
I20260619 22:30:31.000793 27970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:31.007083 27970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172: Bootstrap starting.
I20260619 22:30:31.008044 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:31.000224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48855 (local address 127.5.176.254:39019)
0619 22:30:31.000672 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:31.000711 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:31.000955 (+ 244us) server_negotiation.cc:407] Connection header received
0619 22:30:31.001707 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:31.001724 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:31.002060 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:31.002394 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:31.002723 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:31.003682 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:31.004868 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:31.005348 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:31.005481 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:31.006394 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:31.006400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:31.006412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:31.006816 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:31.006871 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:31.006894 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:31.006975 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:31.007132 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 22:30:31.007344 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":151,"threads_started":1}
I20260619 22:30:31.009462 27970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:31.010347 27970 log.cc:824] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:31.010946 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:31.012295 27970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172: No bootstrap required, opened a new log
I20260619 22:30:31.017097 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:31.012924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52841 (local address 127.5.176.254:39019)
0619 22:30:31.013051 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:31.013055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:31.013241 (+ 186us) server_negotiation.cc:407] Connection header received
0619 22:30:31.013357 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:31.013359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:31.013398 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:31.013461 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:31.013745 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:31.014323 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:31.015504 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:31.015758 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:31.015840 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:31.016639 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:31.016642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:31.016644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:31.016655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:31.016678 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:31.016688 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:31.016744 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:31.016860 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 22:30:31.016906 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:30:31.016939 27970 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1090d78b64e4ff7a700addd62757172" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39019 } }
I20260619 22:30:31.017232 27970 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:31.017292 27970 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1090d78b64e4ff7a700addd62757172, State: Initialized, Role: FOLLOWER
I20260619 22:30:31.017763 27970 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1090d78b64e4ff7a700addd62757172" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39019 } }
I20260619 22:30:31.017916 27970 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:31.018023 27970 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:31.018152 27970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:31.019049 27970 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1090d78b64e4ff7a700addd62757172" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39019 } }
I20260619 22:30:31.019410 27970 leader_election.cc:302] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1090d78b64e4ff7a700addd62757172; no voters:
I20260619 22:30:31.019721 27970 leader_election.cc:288] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:31.019827 27980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:31.020072 27980 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [term 1 LEADER]: Becoming Leader. State: Replica: b1090d78b64e4ff7a700addd62757172, State: Running, Role: LEADER
I20260619 22:30:31.020437 27980 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1090d78b64e4ff7a700addd62757172" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39019 } }
I20260619 22:30:31.020921 27970 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:31.022064 27982 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1090d78b64e4ff7a700addd62757172. Latest consensus state: current_term: 1 leader_uuid: "b1090d78b64e4ff7a700addd62757172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1090d78b64e4ff7a700addd62757172" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39019 } } }
I20260619 22:30:31.022223 27982 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:31.022599 27981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1090d78b64e4ff7a700addd62757172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1090d78b64e4ff7a700addd62757172" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 39019 } } }
I20260619 22:30:31.023355 27981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172 [sys.catalog]: This master's current role is: LEADER
W20260619 22:30:31.025560 27995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:31.025655 27995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:31.025744 27989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:31.026547 27989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:31.032253 27989 catalog_manager.cc:1347] Generated new cluster ID: b733d8bbb21e4011ae8d8cb1d39bc37a
I20260619 22:30:31.032334 27989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:31.054977 27989 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:31.056074 27989 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:31.062302 27989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1090d78b64e4ff7a700addd62757172: Generated new TSK 0
I20260619 22:30:31.063009 27989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:31.072253 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 27902
2026-06-19T22:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:31 dist-test-slave-bt27 krb5kdc[28001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:31 dist-test-slave-bt27 krb5kdc[28001](info): set up 2 sockets
Jun 19 22:30:31 dist-test-slave-bt27 krb5kdc[28001](info): commencing operation
krb5kdc: starting...
W20260619 22:30:32.110551 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:32 dist-test-slave-bt27 krb5kdc[28001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:32.237362 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34265
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:32.349622 28014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:32.349936 28014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:32.349982 28014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:32.354127 28014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:32.354291 28014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:32.354316 28014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:32.354375 28014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:32.354415 28014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:32.359899 28014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:32.361099 28014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:32.362277 28014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:32.371287 28020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:32.371253 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:32.371313 28019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:32.371568 28014 server_base.cc:1034] running on GCE node
I20260619 22:30:32.372223 28014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:32.373327 28014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:32.374511 28014 hybrid_clock.cc:648] HybridClock initialized: now 1781908232374480 us; error 43 us; skew 500 ppm
Jun 19 22:30:32 dist-test-slave-bt27 krb5kdc[28001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:32.377422 28014 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:32.378474 28014 webserver.cc:460] Webserver started at http://127.5.176.254:41137/ using document root <none> and password file <none>
I20260619 22:30:32.379057 28014 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:32.379148 28014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:32.379391 28014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:32.381544 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "a4c72141cc50454fbef0e47fea2543ad"
format_stamp: "Formatted at 2026-06-19 22:30:32 on dist-test-slave-bt27"
server_key: "169f4bdfeca6a1c1809088c6f7dab721"
server_key_iv: "632296860a59edce7f9d613dd7fcd4c0"
server_key_version: "encryptionkey@0"
I20260619 22:30:32.382128 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "a4c72141cc50454fbef0e47fea2543ad"
format_stamp: "Formatted at 2026-06-19 22:30:32 on dist-test-slave-bt27"
server_key: "169f4bdfeca6a1c1809088c6f7dab721"
server_key_iv: "632296860a59edce7f9d613dd7fcd4c0"
server_key_version: "encryptionkey@0"
I20260619 22:30:32.385980 28014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:32.388429 28029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:32.389591 28014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:30:32.389729 28014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "a4c72141cc50454fbef0e47fea2543ad"
format_stamp: "Formatted at 2026-06-19 22:30:32 on dist-test-slave-bt27"
server_key: "169f4bdfeca6a1c1809088c6f7dab721"
server_key_iv: "632296860a59edce7f9d613dd7fcd4c0"
server_key_version: "encryptionkey@0"
I20260619 22:30:32.389840 28014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:32.394691 28014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:32.397776 28014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:32.397982 28014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:32.404907 28014 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:34265
I20260619 22:30:32.404946 28081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:34265 every 8 connection(s)
I20260619 22:30:32.405840 28014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:32.408576 28082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:32.414080 28082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad: Bootstrap starting.
I20260619 22:30:32.415129 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28014
I20260619 22:30:32.415247 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:32.415582 5827 external_mini_cluster.cc:1428] Setting key 3cb561f5c68c8bebaabaa2ecddf09d0b
I20260619 22:30:32.416879 28082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:32.417634 28082 log.cc:824] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:32.419744 28082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad: No bootstrap required, opened a new log
I20260619 22:30:32.420692 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:32.416685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38107 (local address 127.5.176.254:34265)
0619 22:30:32.417199 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:32.417213 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:32.417248 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:32.417986 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:32.418010 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:32.418381 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:32.418748 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:32.419537 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:32.419541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:32.419552 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:32.419562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:32.419613 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:32.419632 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:32.419771 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:32.419858 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 22:30:32.420016 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":145,"threads_started":1}
I20260619 22:30:32.423533 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:32.424067 28082 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c72141cc50454fbef0e47fea2543ad" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34265 } }
I20260619 22:30:32.424314 28082 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 22:30:32.424417 28082 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c72141cc50454fbef0e47fea2543ad, State: Initialized, Role: FOLLOWER
I20260619 22:30:32.424935 28082 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c72141cc50454fbef0e47fea2543ad" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34265 } }
I20260619 22:30:32.425129 28082 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 22:30:32.425200 28082 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 22:30:32.425300 28082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:32.426470 28082 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c72141cc50454fbef0e47fea2543ad" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34265 } }
I20260619 22:30:32.426838 28082 leader_election.cc:302] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4c72141cc50454fbef0e47fea2543ad; no voters:
I20260619 22:30:32.427177 28082 leader_election.cc:288] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 22:30:32.427304 28092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:32.427518 28092 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [term 1 LEADER]: Becoming Leader. State: Replica: a4c72141cc50454fbef0e47fea2543ad, State: Running, Role: LEADER
I20260619 22:30:32.427521 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:32.425719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36551 (local address 127.5.176.254:34265)
0619 22:30:32.425856 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:32.425860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:32.426074 (+ 214us) server_negotiation.cc:407] Connection header received
0619 22:30:32.426248 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:32.426251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:32.426294 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:32.426377 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:32.427097 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:32.427100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:32.427103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 22:30:32.427137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:32.427169 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 22:30:32.427180 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:32.427269 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:32.427359 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 22:30:32.427399 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 22:30:32.428018 28092 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c72141cc50454fbef0e47fea2543ad" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34265 } }
I20260619 22:30:32.428534 28082 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:32.429275 28092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c72141cc50454fbef0e47fea2543ad. Latest consensus state: current_term: 1 leader_uuid: "a4c72141cc50454fbef0e47fea2543ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c72141cc50454fbef0e47fea2543ad" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34265 } } }
I20260619 22:30:32.429410 28092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:32.429805 28094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4c72141cc50454fbef0e47fea2543ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c72141cc50454fbef0e47fea2543ad" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 34265 } } }
I20260619 22:30:32.429914 28094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:32.430440 28100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:32.433671 28100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:32.439175 28100 catalog_manager.cc:1347] Generated new cluster ID: ada1ae7704c847c289be88cfcc4bf7d5
I20260619 22:30:32.439260 28100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:32.459610 28100 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:32.459975 28100 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:32.475909 28100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4c72141cc50454fbef0e47fea2543ad: Generated new TSK 0
I20260619 22:30:32.476732 28100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 22:30:32.494236 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28014
2026-06-19T22:30:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1422 ms)
[----------] 18 tests from AuthTokenIssuingTest (25767 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:32 dist-test-slave-bt27 krb5kdc[28113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:32 dist-test-slave-bt27 krb5kdc[28113](info): set up 2 sockets
Jun 19 22:30:32 dist-test-slave-bt27 krb5kdc[28113](info): commencing operation
krb5kdc: starting...
W20260619 22:30:33.526614 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:33 dist-test-slave-bt27 krb5kdc[28113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:33.651517 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:40447
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:40447,127.5.176.253:37503,127.5.176.252:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:33.764611 28126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:30:33.764941 28126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:33.764968 28126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:33.765009 28126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:33.769331 28126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:33.769415 28126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:33.769438 28126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:33.769459 28126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:33.769477 28126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:33.774947 28126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:40447,127.5.176.253:37503,127.5.176.252:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:33.776031 28126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:33.777047 28126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:33.785890 28134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:33.785890 28131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:33.785930 28132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:33.786037 28126 server_base.cc:1034] running on GCE node
I20260619 22:30:33.786885 28126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:33.788091 28126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:33.789292 28126 hybrid_clock.cc:648] HybridClock initialized: now 1781908233789282 us; error 33 us; skew 500 ppm
Jun 19 22:30:33 dist-test-slave-bt27 krb5kdc[28113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:33.792150 28126 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:33.793081 28126 webserver.cc:460] Webserver started at http://127.5.176.254:34371/ using document root <none> and password file <none>
I20260619 22:30:33.793615 28126 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:33.793673 28126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:33.793838 28126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:33.795926 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "8da08d3c30cf4731aa78c371a8be90d4"
format_stamp: "Formatted at 2026-06-19 22:30:33 on dist-test-slave-bt27"
server_key: "89e84b9e4f955dca5b2133f9a5b714d1"
server_key_iv: "5d7cadf641a078d2e23a2b3c00b5a195"
server_key_version: "encryptionkey@0"
I20260619 22:30:33.796444 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "8da08d3c30cf4731aa78c371a8be90d4"
format_stamp: "Formatted at 2026-06-19 22:30:33 on dist-test-slave-bt27"
server_key: "89e84b9e4f955dca5b2133f9a5b714d1"
server_key_iv: "5d7cadf641a078d2e23a2b3c00b5a195"
server_key_version: "encryptionkey@0"
I20260619 22:30:33.800284 28126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 22:30:33.802821 28141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:33.804100 28126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:30:33.804207 28126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "8da08d3c30cf4731aa78c371a8be90d4"
format_stamp: "Formatted at 2026-06-19 22:30:33 on dist-test-slave-bt27"
server_key: "89e84b9e4f955dca5b2133f9a5b714d1"
server_key_iv: "5d7cadf641a078d2e23a2b3c00b5a195"
server_key_version: "encryptionkey@0"
I20260619 22:30:33.804315 28126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:33.821517 28126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:33.824369 28126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:33.824545 28126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:33.831640 28126 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:40447
I20260619 22:30:33.831919 28193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:40447 every 8 connection(s)
I20260619 22:30:33.832618 28126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:33.835357 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:33.838351 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28126
I20260619 22:30:33.838493 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:33.838759 5827 external_mini_cluster.cc:1428] Setting key a3c261b465bf77e0710b19d38f9d3efb
I20260619 22:30:33.839452 28194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.253@KRBTEST.COM" created.
Jun 19 22:30:33 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:33.856032 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.842066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40447 (local address 127.0.0.1:60272)
0619 22:30:33.842700 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.842747 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:33.843835 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:33.844570 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:33.844579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:33.845078 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:33.845439 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:33.845462 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:33.846686 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:33.846692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:33.847764 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:33.847774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:33.848391 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:33.848394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:33.848483 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:33.849031 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:33.849047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:33.851110 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:33.852991 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:33.853003 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:33.853016 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:33.853329 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:33.853628 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:33.853634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:33.853639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:33.853786 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:33.854470 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:33.854479 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:33.854645 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 22:30:33.854858 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 22:30:33.855255 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":63,"spinlock_wait_cycles":23936,"thread_start_us":185,"threads_started":1}
I20260619 22:30:33.856031 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.842066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.5.176.254:40447)
0619 22:30:33.842598 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:33.842618 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:33.842901 (+ 283us) server_negotiation.cc:407] Connection header received
0619 22:30:33.844171 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:33.844176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:33.844350 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:33.844505 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:33.845615 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:33.846552 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:33.847874 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:33.848227 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:33.848412 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:33.851255 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:33.851260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:33.851269 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:33.851291 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:33.852892 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:33.853439 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:33.853443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:33.853444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:33.853543 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:33.853889 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:33.853891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:33.853892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:33.854355 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:33.854461 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:33.854948 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 22:30:33.855180 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":351,"thread_start_us":130,"threads_started":1}
I20260619 22:30:33.858577 28194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:33.859337 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.859056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50600)
0619 22:30:33.859169 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.859241 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 22:30:33.859553 28142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
I20260619 22:30:33.860406 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.860219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50614)
0619 22:30:33.860317 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.860358 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 22:30:33.860598 28194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:37503: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab.
Entry for principal kudu/127.5.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab.
I20260619 22:30:33.867216 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:37503
--webserver_interface=127.5.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab
--principal=kudu/127.5.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:40447,127.5.176.253:37503,127.5.176.252:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:30:33.909838 28194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } attempt: 1
I20260619 22:30:33.910714 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.910422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50616)
0619 22:30:33.910573 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.910636 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 22:30:33.911383 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.911185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50628)
0619 22:30:33.911283 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.911333 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 22:30:33.911540 28194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:37503: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
W20260619 22:30:33.978896 28201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:30:33.979189 28201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:33.979214 28201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:33.979323 28201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 22:30:33.979692 28194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } attempt: 2
I20260619 22:30:33.980497 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.980210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50638)
0619 22:30:33.980354 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.980425 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:30:33.981220 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:33.981012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50644)
0619 22:30:33.981112 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:33.981157 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 22:30:33.981420 28194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:37503: Network error: Client connection negotiation failed: client connection to 127.5.176.253:37503: connect: Connection refused (error 111)
W20260619 22:30:33.983753 28201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:33.983806 28201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:33.983826 28201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:33.983884 28201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:33.983942 28201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:33.989341 28201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:40447,127.5.176.253:37503,127.5.176.252:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab
--principal=kudu/127.5.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:37503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:33.990504 28201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:33.991534 28201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:34.000041 28209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:34.000078 28207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:34.000068 28206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:34.000419 28201 server_base.cc:1034] running on GCE node
I20260619 22:30:34.000965 28201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:34.002045 28201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:34.003245 28201 hybrid_clock.cc:648] HybridClock initialized: now 1781908234003230 us; error 37 us; skew 500 ppm
Jun 19 22:30:33 dist-test-slave-bt27 krb5kdc[28113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:34.005993 28201 init.cc:377] Logged in from keytab as kudu/127.5.176.253@KRBTEST.COM (short username kudu)
I20260619 22:30:34.006955 28201 webserver.cc:460] Webserver started at http://127.5.176.253:45739/ using document root <none> and password file <none>
I20260619 22:30:34.007480 28201 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:34.007591 28201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:34.007777 28201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:34.009910 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data/instance:
uuid: "11d3ca9ad77146fc839f83254e624dbf"
format_stamp: "Formatted at 2026-06-19 22:30:34 on dist-test-slave-bt27"
server_key: "2f8001df671a07b6621e764101dae5aa"
server_key_iv: "1c60edbe8917428454c0b46dcd61b97f"
server_key_version: "encryptionkey@0"
I20260619 22:30:34.010501 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal/instance:
uuid: "11d3ca9ad77146fc839f83254e624dbf"
format_stamp: "Formatted at 2026-06-19 22:30:34 on dist-test-slave-bt27"
server_key: "2f8001df671a07b6621e764101dae5aa"
server_key_iv: "1c60edbe8917428454c0b46dcd61b97f"
server_key_version: "encryptionkey@0"
I20260619 22:30:34.014324 28201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:30:34.016881 28216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:34.018200 28201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:34.018324 28201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal
uuid: "11d3ca9ad77146fc839f83254e624dbf"
format_stamp: "Formatted at 2026-06-19 22:30:34 on dist-test-slave-bt27"
server_key: "2f8001df671a07b6621e764101dae5aa"
server_key_iv: "1c60edbe8917428454c0b46dcd61b97f"
server_key_version: "encryptionkey@0"
I20260619 22:30:34.018419 28201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:34.029537 28201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:34.032331 28201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:34.032493 28201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:34.039847 28201 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.253:37503
I20260619 22:30:34.040150 28268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.253:37503 every 8 connection(s)
I20260619 22:30:34.040936 28201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
I20260619 22:30:34.043678 28269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:34.043936 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28201
I20260619 22:30:34.044078 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-1/wal/instance
I20260619 22:30:34.044384 5827 external_mini_cluster.cc:1428] Setting key 05aa2bf54d302d9c48345c6b2bf0cf80
I20260619 22:30:34.047674 28269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.176.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.252@KRBTEST.COM" created.
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:34.063088 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.049692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.5.176.254:40447)
0619 22:30:34.049847 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.049851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.050994 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 22:30:34.051934 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.051937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.052003 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.052105 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.053287 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.053867 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.055154 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.055457 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.055572 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.058426 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.058434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.058440 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.058477 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.060843 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.061418 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.061421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.061422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.061498 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.061875 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.061878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.061879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.062066 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.062198 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.062766 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.062908 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:30:34.063910 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.050278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40447 (local address 127.0.0.1:60282)
0619 22:30:34.050844 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.050885 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.051714 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.052275 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.052282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.052661 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.053143 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.053156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.053995 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.054003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.055026 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.055036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.055616 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.055620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.055800 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.056428 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.056444 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.058231 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.060982 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.060989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.061012 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.061298 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.061601 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.061609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.061610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.061755 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.062199 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.062207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.062501 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.062699 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.063051 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":168,"threads_started":1}
I20260619 22:30:34.065459 28269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
I20260619 22:30:34.072593 28194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab.
Entry for principal kudu/127.5.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab.
I20260619 22:30:34.073971 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:38367
--webserver_interface=127.5.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab
--principal=kudu/127.5.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:40447,127.5.176.253:37503,127.5.176.252:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:30:34.075527 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.065993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50656)
0619 22:30:34.066114 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.066128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.066224 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.066882 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.066887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.067046 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.067134 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.067139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.068368 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.068371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.068996 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.069005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.069695 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.069698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.069804 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.070361 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.070382 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.071686 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.073839 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.073842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.073843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.074020 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.074498 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.074504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.074506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.074595 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.075079 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.075083 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.075166 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.075238 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.075360 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 22:30:34.076396 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.066208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.5.176.253:37503)
0619 22:30:34.066533 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.066541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.066561 (+ 20us) server_negotiation.cc:407] Connection header received
0619 22:30:34.066596 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.066600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.066746 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.066860 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.067230 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.068241 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.069094 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.069591 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.069678 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.071809 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.071818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.071830 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.071851 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.073726 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.074282 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.074291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.074296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.074391 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.074690 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.074697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.074699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.074952 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.075876 (+ 924us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.076000 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.076165 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":150,"threads_started":1}
I20260619 22:30:34.078743 28269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:34.079563 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.079217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54508)
0619 22:30:34.079376 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.079471 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
W20260619 22:30:34.079785 28218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
I20260619 22:30:34.080847 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.080531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54516)
0619 22:30:34.080685 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.080740 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":43}
W20260619 22:30:34.081003 28269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:38367: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
I20260619 22:30:34.083666 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.073122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.5.176.253:37503)
0619 22:30:34.074351 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.074358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.074368 (+ 10us) server_negotiation.cc:407] Connection header received
0619 22:30:34.074395 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.074396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.074428 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.074531 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.075036 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.075939 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.076855 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.077108 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.077212 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.080246 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.080251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.080253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.080279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.081929 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.082478 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.082481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.082482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.082560 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.082887 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.082889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.082890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.083055 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.083126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.083466 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.083555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":98,"threads_started":1}
I20260619 22:30:34.083842 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.073088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50660)
0619 22:30:34.073400 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.073415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.073486 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.074566 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.074572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.074769 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.074909 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.074920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.076049 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.076053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.076736 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.076747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.077228 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.077232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.077353 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.078510 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.078531 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.080058 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.082046 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.082052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.082056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.082373 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.082657 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.082663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.082668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.082789 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.083152 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.083156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.083294 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.083423 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.083595 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260619 22:30:34.084668 28194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:34.085386 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.085166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54530)
0619 22:30:34.085277 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.085328 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 22:30:34.086185 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.085922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54540)
0619 22:30:34.086061 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.086099 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260619 22:30:34.086333 28194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:38367: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
I20260619 22:30:34.117532 28194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } attempt: 1
I20260619 22:30:34.118479 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.118179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54554)
0619 22:30:34.118338 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.118402 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:30:34.119309 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.119100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54562)
0619 22:30:34.119201 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.119246 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 22:30:34.119475 28194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:38367: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
I20260619 22:30:34.130245 28269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } attempt: 1
I20260619 22:30:34.131192 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54572)
0619 22:30:34.131026 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.131098 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 22:30:34.132025 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.131804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54588)
0619 22:30:34.131913 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.131964 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 22:30:34.132159 28269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:38367: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
W20260619 22:30:34.191051 28276 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:30:34.191367 28276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:34.191393 28276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:34.191433 28276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 22:30:34.194654 28194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } attempt: 2
I20260619 22:30:34.195583 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.195255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54604)
0619 22:30:34.195435 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.195501 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 22:30:34.195960 28276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:34.196044 28276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:34.196087 28276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:34.196127 28276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:34.196166 28276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:34.196470 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.196246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54606)
0619 22:30:34.196360 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.196411 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:30:34.196678 28194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:38367: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
I20260619 22:30:34.200323 28269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } attempt: 2
I20260619 22:30:34.201077 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.200776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54608)
0619 22:30:34.200935 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.200993 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 22:30:34.201850 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.201615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54612)
0619 22:30:34.201728 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.201780 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 22:30:34.202064 28269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:38367: Network error: Client connection negotiation failed: client connection to 127.5.176.252:38367: connect: Connection refused (error 111)
I20260619 22:30:34.202575 28276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:40447,127.5.176.253:37503,127.5.176.252:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab
--principal=kudu/127.5.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:34.203696 28276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:34.204751 28276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:34.213941 28282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:34.214004 28276 server_base.cc:1034] running on GCE node
W20260619 22:30:34.213946 28283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:34.213977 28285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:34.214934 28276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:34.216169 28276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:34.217391 28276 hybrid_clock.cc:648] HybridClock initialized: now 1781908234217363 us; error 43 us; skew 500 ppm
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908234, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:34.220352 28276 init.cc:377] Logged in from keytab as kudu/127.5.176.252@KRBTEST.COM (short username kudu)
I20260619 22:30:34.221385 28276 webserver.cc:460] Webserver started at http://127.5.176.252:32953/ using document root <none> and password file <none>
I20260619 22:30:34.221938 28276 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:34.221995 28276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:34.222234 28276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:34.224385 28276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data/instance:
uuid: "4b4de5873a54449ca84820a1bb54b0d7"
format_stamp: "Formatted at 2026-06-19 22:30:34 on dist-test-slave-bt27"
server_key: "f734d74e5243ba69bbf39f70b1918bec"
server_key_iv: "304a01369ff972804ace85592f49a1de"
server_key_version: "encryptionkey@0"
I20260619 22:30:34.224947 28276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal/instance:
uuid: "4b4de5873a54449ca84820a1bb54b0d7"
format_stamp: "Formatted at 2026-06-19 22:30:34 on dist-test-slave-bt27"
server_key: "f734d74e5243ba69bbf39f70b1918bec"
server_key_iv: "304a01369ff972804ace85592f49a1de"
server_key_version: "encryptionkey@0"
I20260619 22:30:34.228708 28276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 22:30:34.231194 28292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:34.232466 28276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:34.232616 28276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal
uuid: "4b4de5873a54449ca84820a1bb54b0d7"
format_stamp: "Formatted at 2026-06-19 22:30:34 on dist-test-slave-bt27"
server_key: "f734d74e5243ba69bbf39f70b1918bec"
server_key_iv: "304a01369ff972804ace85592f49a1de"
server_key_version: "encryptionkey@0"
I20260619 22:30:34.232744 28276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:34.251204 28276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:34.254828 28276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:34.254994 28276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:34.262128 28276 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.252:38367
I20260619 22:30:34.262463 28344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.252:38367 every 8 connection(s)
I20260619 22:30:34.263264 28276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
I20260619 22:30:34.266116 28345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:34.270188 28345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:34.272013 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28276
I20260619 22:30:34.272168 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781907951519384-5827-0/minicluster-data/master-2/wal/instance
I20260619 22:30:34.272478 5827 external_mini_cluster.cc:1428] Setting key dd1efd647869904391d9b55a9bbba1c6
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908234, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
I20260619 22:30:34.284520 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.272462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.5.176.254:40447)
0619 22:30:34.272623 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.272627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.273583 (+ 956us) server_negotiation.cc:407] Connection header received
0619 22:30:34.274744 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.274747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.274814 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.274914 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.275749 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.276258 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.277254 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.277468 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.277578 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.280343 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.280347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.280356 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.280381 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.282013 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.282639 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.282643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.282644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.282747 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.283112 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.283118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.283123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.283397 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.283504 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.284169 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.284329 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:30:34.285539 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.272881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:40447 (local address 127.0.0.1:60294)
0619 22:30:34.273408 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.273464 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.274515 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.274962 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.274967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.275316 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.275601 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.275614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.276368 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.276373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.277124 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.277132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.277615 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.277619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.277872 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.278530 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.278553 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.280197 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.282129 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.282154 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.282167 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.282525 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.282860 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.282863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.282867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.283005 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.283525 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.283532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.283904 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.284075 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.284441 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":47,"thread_start_us":169,"threads_started":1}
I20260619 22:30:34.285539 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.273832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.5.176.252:38367)
0619 22:30:34.274123 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.274131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.274256 (+ 125us) server_negotiation.cc:407] Connection header received
0619 22:30:34.274525 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.274545 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.274764 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.274959 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.275814 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.276663 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.277511 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.277907 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.278016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.280740 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.280748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.280764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.280793 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.282736 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.283169 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.283173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.283174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.283239 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.283525 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.283528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.283530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.283920 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.284129 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.284422 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.284541 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":197,"spinlock_wait_cycles":3840,"thread_start_us":93,"threads_started":1}
I20260619 22:30:34.287117 28345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908234, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
I20260619 22:30:34.293251 28269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } attempt: 3
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:34.295830 28194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } attempt: 3
I20260619 22:30:34.297425 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.287755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.5.176.253:37503)
0619 22:30:34.287873 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.287877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.287893 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:30:34.287944 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.287947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.287997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.288084 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.288555 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.289437 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.290429 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.290885 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.290996 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.293310 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.293316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.293325 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.293349 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.295453 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.296002 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.296005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.296007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.296097 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.296441 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.296445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.296447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.296688 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.296744 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.297103 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.297237 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 22:30:34.297941 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.287643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:37503 (local address 127.0.0.1:50676)
0619 22:30:34.287768 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.287782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.287865 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.288097 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.288100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.288289 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.288417 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.288426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.289541 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.289544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.290326 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.290334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.290996 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.290998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.291080 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.291683 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.291700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.293137 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.295571 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.295574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.295576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.295890 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.296212 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.296215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.296217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.296310 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.296802 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.296804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.296919 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.297614 (+ 695us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.297781 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 22:30:34.299113 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.288288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.5.176.254:40447)
0619 22:30:34.288410 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.288414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.288427 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:30:34.288488 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.288493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.288543 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.288610 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.289341 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.290031 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.291145 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.291428 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.291538 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.294478 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.294482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.294485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.294531 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.296528 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.297044 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.297047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.297049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.297144 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.297465 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.297468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.297470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.297904 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.298164 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.298818 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.298938 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":51}
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
I20260619 22:30:34.299443 28345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
I20260619 22:30:34.305101 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.293819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54624)
0619 22:30:34.293948 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.293963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.294045 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.294343 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.294348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.294518 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.294621 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.294627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.295731 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.295738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.296525 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.296534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.297444 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.297451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.297544 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.298006 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.298020 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.299550 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.302812 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.302815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.302818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.303072 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.303371 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.303374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.303376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.303462 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.304450 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.304453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.304633 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.304755 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.304919 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":70}
I20260619 22:30:34.305269 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.293894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.5.176.252:38367)
0619 22:30:34.294036 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.294043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.294060 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:30:34.294157 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.294160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.294214 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.294304 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.294793 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.295614 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.296947 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.297313 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.299848 (+ 2535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.299959 (+ 111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.299962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.299965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.299981 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.302694 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.303186 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.303189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.303191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.303270 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.303562 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.303565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.303567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.303786 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.304523 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.304994 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.305104 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":56}
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908234, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
Jun 19 22:30:34 dist-test-slave-bt27 krb5kdc[28113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
I20260619 22:30:34.308300 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.296253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54628)
0619 22:30:34.296443 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.296458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.296528 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.297342 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.297346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.297514 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.297685 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.297699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.299127 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.299133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.299926 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.299937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.301522 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.301526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.301613 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.302014 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.302030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.303566 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.306359 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.306363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.306365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.306637 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.306983 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.306989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.306995 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.307096 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.307687 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.307693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.307815 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.307908 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.308071 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":19}
I20260619 22:30:34.308910 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.296706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.5.176.252:38367)
0619 22:30:34.297100 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.297110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.297121 (+ 11us) server_negotiation.cc:407] Connection header received
0619 22:30:34.297151 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.297153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.297210 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.297894 (+ 684us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.297981 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.299016 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.300714 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.301396 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.301834 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.303703 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.303706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.303709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.303732 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.306225 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.306772 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.306776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.306778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.306869 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.307239 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.307243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.307245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.307547 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.308530 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.308614 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.308739 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":281,"server-negotiator.queue_time_us":325,"thread_start_us":123,"threads_started":1}
I20260619 22:30:34.309459 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: Bootstrap starting.
I20260619 22:30:34.310689 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.299933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:38367 (local address 127.0.0.1:54630)
0619 22:30:34.300375 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:34.300390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:34.300463 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:34.300861 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:34.300867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:34.301043 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:34.301116 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.301121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.302031 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.302034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.303060 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:34.303071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.303703 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.303706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:34.303794 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.304206 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:34.304217 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:34.306360 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:34.309148 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.309151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.309153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.309338 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.309644 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:34.309647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:34.309649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:34.309729 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:34.310092 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:34.310095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:34.310221 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 22:30:34.310328 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 22:30:34.310488 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
I20260619 22:30:34.310793 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.300103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.5.176.252:38367)
0619 22:30:34.300404 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.300412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.300425 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:30:34.300567 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.300570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.300612 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.300709 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.301217 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.301930 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.303183 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.303554 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.303876 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.306482 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.306485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.306487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.306507 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.308299 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.309417 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.309419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.309420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.309469 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.309822 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.309827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.309829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.309991 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.310094 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.310408 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.310556 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":228,"server-negotiator.queue_time_us":233,"thread_start_us":139,"threads_started":1}
I20260619 22:30:34.311872 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.300628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.5.176.253:37503)
0619 22:30:34.300775 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.300782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.300898 (+ 116us) server_negotiation.cc:407] Connection header received
0619 22:30:34.301010 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.301015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.301066 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.301153 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.302070 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.302867 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.303952 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.304338 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.304444 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.308123 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.308127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.308130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.308155 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.309741 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.310233 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.310236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.310238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.310324 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.310642 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.310645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.310647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.310891 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.311045 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.311583 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.311700 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":60}
I20260619 22:30:34.313333 28194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: Bootstrap starting.
I20260619 22:30:34.313333 28345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: Bootstrap starting.
I20260619 22:30:34.316232 28194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:34.316218 28345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:34.316512 28269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:34.317142 28194 log.cc:824] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:34.317142 28345 log.cc:824] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:34.317370 28269 log.cc:824] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:34.317751 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 22:30:34.319650 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: No bootstrap required, opened a new log
I20260619 22:30:34.319650 28194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: No bootstrap required, opened a new log
I20260619 22:30:34.319846 28345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: No bootstrap required, opened a new log
I20260619 22:30:34.319298 28168 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8da08d3c30cf4731aa78c371a8be90d4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298
I20260619 22:30:34.322830 28194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.323087 28194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da08d3c30cf4731aa78c371a8be90d4, State: Initialized, Role: FOLLOWER
I20260619 22:30:34.323457 28194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.323843 28269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.324033 28269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d3ca9ad77146fc839f83254e624dbf, State: Initialized, Role: FOLLOWER
I20260619 22:30:34.324097 28355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.324332 28355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.324311 28269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.324688 28345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.324937 28345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4de5873a54449ca84820a1bb54b0d7, State: Initialized, Role: FOLLOWER
I20260619 22:30:34.324998 28194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:34.325073 28168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 22:30:34.325176 28168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:34.325223 28345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.325554 28269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: configured and running, proceeding with master startup.
W20260619 22:30:34.326025 28147 tablet.cc:2369] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:30:34.326699 28345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:34.327178 28168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.330009 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8da08d3c30cf4731aa78c371a8be90d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b4de5873a54449ca84820a1bb54b0d7"
I20260619 22:30:34.330257 28319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:34.331664 28319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da08d3c30cf4731aa78c371a8be90d4 in term 1.
W20260619 22:30:34.332000 28390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:34.332077 28390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 22:30:34.332057 28389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:34.332283 28389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:34.332952 28143 leader_election.cc:302] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4de5873a54449ca84820a1bb54b0d7, 8da08d3c30cf4731aa78c371a8be90d4; no voters:
W20260619 22:30:34.332989 28378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:34.333097 28378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:34.333233 28355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:34.333442 28355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [term 1 LEADER]: Becoming Leader. State: Replica: 8da08d3c30cf4731aa78c371a8be90d4, State: Running, Role: LEADER
I20260619 22:30:34.333573 28168 leader_election.cc:288] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [CANDIDATE]: Term 1 election: Requested vote from peers 11d3ca9ad77146fc839f83254e624dbf (127.5.176.253:37503), 4b4de5873a54449ca84820a1bb54b0d7 (127.5.176.252:38367)
I20260619 22:30:34.333760 28355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } }
I20260619 22:30:34.329049 28243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8da08d3c30cf4731aa78c371a8be90d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "11d3ca9ad77146fc839f83254e624dbf"
I20260619 22:30:34.334017 28243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:34.335286 28243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da08d3c30cf4731aa78c371a8be90d4 in term 1.
I20260619 22:30:34.335587 28355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da08d3c30cf4731aa78c371a8be90d4. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.335706 28355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:34.345966 28126 master_runner.cc:263] No leader master found from master 8da08d3c30cf4731aa78c371a8be90d4
I20260619 22:30:34.346063 28126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:30:34.324908 28356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.346292 28356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.349959 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.338905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.5.176.254:40447)
0619 22:30:34.339014 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.339019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.339626 (+ 607us) server_negotiation.cc:407] Connection header received
0619 22:30:34.339836 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.339840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.339895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.339947 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.340996 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.341803 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.343047 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.343361 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.343439 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.345834 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.345839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.345842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.345872 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.347755 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.348291 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.348294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.348296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.348400 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.348721 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.348725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.348743 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.348976 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.349080 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.349553 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.349755 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 22:30:34.350642 28396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:34.351836 28396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:34.355370 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.344775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.5.176.253:37503)
0619 22:30:34.344927 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.344932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.345599 (+ 667us) server_negotiation.cc:407] Connection header received
0619 22:30:34.345810 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.345813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.345862 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.345917 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.347091 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.347816 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.348980 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.349290 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.349372 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.351640 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.351644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.351647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.351676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.353398 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.353904 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.353908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.353911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.354001 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.354352 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.354355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.354357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.354577 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.354700 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.355080 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.355209 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 22:30:34.357661 28243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [term 1 FOLLOWER]: Refusing update from remote peer 8da08d3c30cf4731aa78c371a8be90d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:34.357676 28319 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [term 1 FOLLOWER]: Refusing update from remote peer 8da08d3c30cf4731aa78c371a8be90d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:34.358340 28355 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:34.358757 28355 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:34.365190 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:34.346696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.5.176.252:38367)
0619 22:30:34.346815 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.346820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.349987 (+ 3167us) server_negotiation.cc:407] Connection header received
0619 22:30:34.350208 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.350211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.350272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.350331 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.351253 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.352054 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.359158 (+ 7104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:34.359461 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:34.359524 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:34.361276 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:34.361280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:34.361283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:34.361311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:34.363113 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.363614 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.363617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.363619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.363705 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:34.364020 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:34.364023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:34.364025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:34.364270 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:34.364439 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:34.364902 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 22:30:34.365029 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 22:30:34.365736 28356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da08d3c30cf4731aa78c371a8be90d4. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.365886 28356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.368146 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.368312 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.368413 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da08d3c30cf4731aa78c371a8be90d4. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.368484 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.369807 28404 mvcc.cc:205] Tried to move back new op lower bound from 7298696127921860608 to 7298696127834710016. Current Snapshot: MvccSnapshot[applied={T|T < 7298696127921860608}]
I20260619 22:30:34.369930 28394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.370074 28394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:34.370101 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.370218 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.372283 28355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.372406 28355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:34.373306 28356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8da08d3c30cf4731aa78c371a8be90d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da08d3c30cf4731aa78c371a8be90d4" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 40447 } } peers { permanent_uuid: "11d3ca9ad77146fc839f83254e624dbf" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 37503 } } peers { permanent_uuid: "4b4de5873a54449ca84820a1bb54b0d7" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 38367 } } }
I20260619 22:30:34.373438 28356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:34.376879 28405 mvcc.cc:205] Tried to move back new op lower bound from 7298696127921860608 to 7298696127834710016. Current Snapshot: MvccSnapshot[applied={T|T < 7298696127921860608}]
I20260619 22:30:34.377152 28408 mvcc.cc:205] Tried to move back new op lower bound from 7298696127921860608 to 7298696127834710016. Current Snapshot: MvccSnapshot[applied={T|T < 7298696127921860608}]
I20260619 22:30:34.379716 28396 catalog_manager.cc:1347] Generated new cluster ID: 785f512b3e724652bb2ad77595052b8b
I20260619 22:30:34.379797 28396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:34.421125 28396 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:34.422958 28396 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:34.449294 28396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da08d3c30cf4731aa78c371a8be90d4: Generated new TSK 0
I20260619 22:30:34.450508 28396 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:30:34.496316 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.485646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.5.176.253:37503)
0619 22:30:34.485795 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.485801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.494572 (+ 8771us) server_negotiation.cc:407] Connection header received
0619 22:30:34.495259 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.495263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.495328 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.495388 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.496170 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50688: BlockingRecv error: recv got EOF from 127.0.0.1:50688 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 22:30:34.498668 28351 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.495106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.5.176.252:38367)
0619 22:30:34.495267 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.495273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.497624 (+ 2351us) server_negotiation.cc:407] Connection header received
0619 22:30:34.497809 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.497813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.497863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.497915 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.498541 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54638: BlockingRecv error: recv got EOF from 127.0.0.1:54638 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 22:30:34.499068 28197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.497063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.5.176.254:40447)
0619 22:30:34.497207 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.497212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.498040 (+ 828us) server_negotiation.cc:407] Connection header received
0619 22:30:34.498245 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.498249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.498303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.498354 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.498946 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60318: BlockingRecv error: recv got EOF from 127.0.0.1:60318 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 22:30:34.520418 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.518666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.5.176.253:37503)
0619 22:30:34.518825 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.518830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.519233 (+ 403us) server_negotiation.cc:407] Connection header received
0619 22:30:34.519505 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.519509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.519590 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.519658 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.520272 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50704: BlockingRecv error: recv got EOF from 127.0.0.1:50704 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 22:30:34.536094 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.524214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.5.176.253:37503)
0619 22:30:34.524342 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.524347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.535111 (+ 10764us) server_negotiation.cc:407] Connection header received
0619 22:30:34.535281 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.535285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.535366 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.535474 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.535976 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50712: BlockingRecv error: recv got EOF from 127.0.0.1:50712 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 22:30:34.551823 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.540330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.5.176.253:37503)
0619 22:30:34.540439 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.540443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.550621 (+ 10178us) server_negotiation.cc:407] Connection header received
0619 22:30:34.550900 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.550904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.550964 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.551021 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.551672 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50722: BlockingRecv error: recv got EOF from 127.0.0.1:50722 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 22:30:34.567771 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.558807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.5.176.253:37503)
0619 22:30:34.558970 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.558975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.566526 (+ 7551us) server_negotiation.cc:407] Connection header received
0619 22:30:34.566753 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.566756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.566814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.566868 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.567637 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50734: BlockingRecv error: recv got EOF from 127.0.0.1:50734 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 22:30:34.582950 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.581578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.5.176.253:37503)
0619 22:30:34.581707 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.581712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.581983 (+ 271us) server_negotiation.cc:407] Connection header received
0619 22:30:34.582119 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.582123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.582215 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.582322 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.582807 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50750: BlockingRecv error: recv got EOF from 127.0.0.1:50750 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 22:30:34.602684 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.601395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.5.176.253:37503)
0619 22:30:34.601524 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.601529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.601728 (+ 199us) server_negotiation.cc:407] Connection header received
0619 22:30:34.601853 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.601856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.601914 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.602015 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.602563 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50766: BlockingRecv error: recv got EOF from 127.0.0.1:50766 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 22:30:34.638526 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.637382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.5.176.253:37503)
0619 22:30:34.637512 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.637516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.637738 (+ 222us) server_negotiation.cc:407] Connection header received
0619 22:30:34.637864 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.637867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.637914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.637993 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.638428 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50778: BlockingRecv error: recv got EOF from 127.0.0.1:50778 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 22:30:34.706117 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.704885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.5.176.253:37503)
0619 22:30:34.705044 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.705049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.705204 (+ 155us) server_negotiation.cc:407] Connection header received
0619 22:30:34.705403 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.705406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.705452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.705547 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.706009 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50784: BlockingRecv error: recv got EOF from 127.0.0.1:50784 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 22:30:34.837657 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:34.836374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.5.176.253:37503)
0619 22:30:34.836516 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:34.836521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:34.836678 (+ 157us) server_negotiation.cc:407] Connection header received
0619 22:30:34.836796 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:34.836799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:34.836864 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:34.836970 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:34.837521 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50794: BlockingRecv error: recv got EOF from 127.0.0.1:50794 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 22:30:35.097153 28275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:35.095924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.5.176.253:37503)
0619 22:30:35.096088 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:35.096093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:35.096185 (+ 92us) server_negotiation.cc:407] Connection header received
0619 22:30:35.096363 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:35.096366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:35.096430 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:35.096527 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:35.097001 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50796: BlockingRecv error: recv got EOF from 127.0.0.1:50796 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 22:30:35.333431 28390 catalog_manager.cc:1259] Loaded cluster ID: 785f512b3e724652bb2ad77595052b8b
I20260619 22:30:35.333516 28390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: loading cluster ID for follower catalog manager: success
I20260619 22:30:35.333529 28389 catalog_manager.cc:1259] Loaded cluster ID: 785f512b3e724652bb2ad77595052b8b
I20260619 22:30:35.333600 28389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: loading cluster ID for follower catalog manager: success
I20260619 22:30:35.335299 28390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: acquiring CA information for follower catalog manager: success
I20260619 22:30:35.335305 28389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: acquiring CA information for follower catalog manager: success
I20260619 22:30:35.336468 28389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11d3ca9ad77146fc839f83254e624dbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:30:35.336468 28390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b4de5873a54449ca84820a1bb54b0d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:30:35.615217 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:35.611312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.5.176.253:37503)
0619 22:30:35.611571 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:35.611575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:35.611591 (+ 16us) server_negotiation.cc:407] Connection header received
0619 22:30:35.611742 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:35.611746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:35.611926 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:35.612048 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:35.612300 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.612819 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.613722 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.613942 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.614012 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:35.614430 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:35.614878 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:35.614965 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:35.615044 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 22:30:35.615091 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20260619 22:30:35.622325 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:35.618507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.5.176.252:38367)
0619 22:30:35.618747 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:35.618753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:35.618770 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:30:35.618885 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:35.618887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:35.619002 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:35.619127 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:35.619354 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.619880 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.620720 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.620957 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.621064 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:35.621432 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:35.621886 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:35.622023 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:35.622060 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 22:30:35.622119 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":57,"threads_started":1}
I20260619 22:30:35.630075 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:35.625653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.5.176.253:37503)
0619 22:30:35.625784 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:35.625789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:35.626024 (+ 235us) server_negotiation.cc:407] Connection header received
0619 22:30:35.626180 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:35.626183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:35.626236 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:35.626324 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:35.626544 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.627408 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.628699 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.629088 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.629183 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:35.629540 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:35.629625 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:35.629672 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:35.629842 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 22:30:35.629906 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 22:30:35.630312 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:35.625433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.5.176.254:40447)
0619 22:30:35.625771 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:35.625776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:35.625791 (+ 15us) server_negotiation.cc:407] Connection header received
0619 22:30:35.626010 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:35.626018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:35.626246 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:35.626383 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:35.626613 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.627435 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.628500 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.628854 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.629017 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:35.629357 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:35.629656 (+ 299us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:35.629759 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:35.630069 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 22:30:35.630161 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":72,"threads_started":1}
I20260619 22:30:35.630635 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:35.625814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.5.176.252:38367)
0619 22:30:35.625915 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:35.625919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:35.626656 (+ 737us) server_negotiation.cc:407] Connection header received
0619 22:30:35.626808 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:35.626811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:35.626860 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:35.626961 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:35.627217 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.628004 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.629253 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:35.629621 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:35.629705 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:35.630111 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:35.630218 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:35.630271 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:35.630451 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 22:30:35.630505 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 22:30:35.637086 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28126
I20260619 22:30:35.644423 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28201
I20260619 22:30:35.650928 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28276
2026-06-19T22:30:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3158 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 22:30:35 dist-test-slave-bt27 krb5kdc[28496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 22:30:35 dist-test-slave-bt27 krb5kdc[28496](info): set up 2 sockets
Jun 19 22:30:35 dist-test-slave-bt27 krb5kdc[28496](info): commencing operation
krb5kdc: starting...
W20260619 22:30:36.702566 5827 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/test-user.keytab.
Jun 19 22:30:36 dist-test-slave-bt27 krb5kdc[28496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T22:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T22:30:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
Entry for principal kudu/127.5.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab.
I20260619 22:30:36.830044 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:33937
--webserver_interface=127.5.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:33937,127.5.176.253:45903,127.5.176.252:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 22:30:36.942685 28509 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:30:36.942982 28509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:36.943006 28509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:36.943100 28509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:36.947154 28509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:36.947214 28509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:36.947234 28509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:36.947252 28509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:36.947314 28509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:36.952620 28509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:33937,127.5.176.253:45903,127.5.176.252:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.254.keytab
--principal=kudu/127.5.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.254:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:36.953694 28509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:36.954772 28509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:36.963732 28514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:36.963801 28515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:36.963851 28509 server_base.cc:1034] running on GCE node
W20260619 22:30:36.963732 28517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:36.964617 28509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:36.965754 28509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:36.966924 28509 hybrid_clock.cc:648] HybridClock initialized: now 1781908236966906 us; error 32 us; skew 500 ppm
Jun 19 22:30:36 dist-test-slave-bt27 krb5kdc[28496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:36.969782 28509 init.cc:377] Logged in from keytab as kudu/127.5.176.254@KRBTEST.COM (short username kudu)
I20260619 22:30:36.970783 28509 webserver.cc:460] Webserver started at http://127.5.176.254:41275/ using document root <none> and password file <none>
I20260619 22:30:36.971385 28509 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:36.971449 28509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:36.971681 28509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:36.973821 28509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data/instance:
uuid: "90dd1260ba514226a61e454611d6db46"
format_stamp: "Formatted at 2026-06-19 22:30:36 on dist-test-slave-bt27"
server_key: "0b21380596446a291485b70f20bb26a5"
server_key_iv: "ff81c55cc6ab0a809de896f575325cc7"
server_key_version: "encryptionkey@0"
I20260619 22:30:36.974526 28509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal/instance:
uuid: "90dd1260ba514226a61e454611d6db46"
format_stamp: "Formatted at 2026-06-19 22:30:36 on dist-test-slave-bt27"
server_key: "0b21380596446a291485b70f20bb26a5"
server_key_iv: "ff81c55cc6ab0a809de896f575325cc7"
server_key_version: "encryptionkey@0"
I20260619 22:30:36.978461 28509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 22:30:36.980880 28524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:36.982103 28509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 22:30:36.982311 28509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal
uuid: "90dd1260ba514226a61e454611d6db46"
format_stamp: "Formatted at 2026-06-19 22:30:36 on dist-test-slave-bt27"
server_key: "0b21380596446a291485b70f20bb26a5"
server_key_iv: "ff81c55cc6ab0a809de896f575325cc7"
server_key_version: "encryptionkey@0"
I20260619 22:30:36.982431 28509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:37.011695 28509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:37.015241 28509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:37.015465 28509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:37.022893 28509 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.254:33937
I20260619 22:30:37.023190 28576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.254:33937 every 8 connection(s)
I20260619 22:30:37.023927 28509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/data/info.pb
I20260619 22:30:37.026816 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28509
I20260619 22:30:37.026844 28577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:37.026964 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-0/wal/instance
I20260619 22:30:37.027261 5827 external_mini_cluster.cc:1428] Setting key 210b122fbc6e40033eaf9d250a910c8f
I20260619 22:30:37.030691 28577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.253@KRBTEST.COM" created.
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:37.047431 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.032964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33937 (local address 127.0.0.1:34886)
0619 22:30:37.033471 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.033525 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.034474 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.035298 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.035303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.035628 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.035882 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.035894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.037086 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.037093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.037966 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.037980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.038925 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.038931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.039094 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.039719 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.039746 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.041466 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.044066 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.044074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.044090 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.044346 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.044762 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.044765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.044766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.044885 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.045732 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.045738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.045888 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.046048 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.046594 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":144,"thread_start_us":150,"threads_started":1}
I20260619 22:30:37.047437 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.032965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.5.176.254:33937)
0619 22:30:37.033536 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.033548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.033635 (+ 87us) server_negotiation.cc:407] Connection header received
0619 22:30:37.034805 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.034811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.035084 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.035298 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.036018 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.036921 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.038104 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.038717 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.038869 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.041636 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.041647 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.041665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.041697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.043926 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.044482 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.044490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.044493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.044646 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.044985 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.044990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.044994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.045585 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.045740 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.046203 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.046518 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":439,"thread_start_us":146,"threads_started":1}
I20260619 22:30:37.050042 28577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.050827 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.050561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40462)
0619 22:30:37.050686 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.050766 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 22:30:37.051091 28527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
I20260619 22:30:37.051921 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.051719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40470)
0619 22:30:37.051823 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.051864 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 22:30:37.052170 28577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:45903: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab.
Entry for principal kudu/127.5.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab.
I20260619 22:30:37.055545 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:45903
--webserver_interface=127.5.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab
--principal=kudu/127.5.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:33937,127.5.176.253:45903,127.5.176.252:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:30:37.101445 28577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } attempt: 1
I20260619 22:30:37.102515 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.102192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40472)
0619 22:30:37.102365 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.102433 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 22:30:37.103399 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.103080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40484)
0619 22:30:37.103187 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.103243 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:30:37.103564 28577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:45903: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
W20260619 22:30:37.170399 28584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:30:37.170755 28584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:37.170819 28584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:37.170912 28584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 22:30:37.171754 28577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } attempt: 2
I20260619 22:30:37.172605 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.172297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40498)
0619 22:30:37.172456 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.172525 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 22:30:37.173331 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.173111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40508)
0619 22:30:37.173213 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.173267 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:30:37.173475 28577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.253:45903: Network error: Client connection negotiation failed: client connection to 127.5.176.253:45903: connect: Connection refused (error 111)
W20260619 22:30:37.175542 28584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:37.175601 28584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:37.175621 28584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:37.175681 28584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:37.175735 28584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:37.181383 28584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:33937,127.5.176.253:45903,127.5.176.252:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.253.keytab
--principal=kudu/127.5.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.253:45903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:37.182591 28584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:37.183624 28584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:37.192353 28590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:37.192413 28584 server_base.cc:1034] running on GCE node
W20260619 22:30:37.192377 28589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:37.192353 28592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:37.193161 28584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:37.194387 28584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:37.195583 28584 hybrid_clock.cc:648] HybridClock initialized: now 1781908237195548 us; error 50 us; skew 500 ppm
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:37.198493 28584 init.cc:377] Logged in from keytab as kudu/127.5.176.253@KRBTEST.COM (short username kudu)
I20260619 22:30:37.199414 28584 webserver.cc:460] Webserver started at http://127.5.176.253:38307/ using document root <none> and password file <none>
I20260619 22:30:37.199993 28584 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:37.200088 28584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:37.200336 28584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:37.202592 28584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data/instance:
uuid: "f12059c2e8e84a0a9c934b87eb31254b"
format_stamp: "Formatted at 2026-06-19 22:30:37 on dist-test-slave-bt27"
server_key: "b7f011bc4b05bc8bbb63263e5528bb3a"
server_key_iv: "26463e4a29b9333137a542c90ac043b3"
server_key_version: "encryptionkey@0"
I20260619 22:30:37.203167 28584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal/instance:
uuid: "f12059c2e8e84a0a9c934b87eb31254b"
format_stamp: "Formatted at 2026-06-19 22:30:37 on dist-test-slave-bt27"
server_key: "b7f011bc4b05bc8bbb63263e5528bb3a"
server_key_iv: "26463e4a29b9333137a542c90ac043b3"
server_key_version: "encryptionkey@0"
I20260619 22:30:37.207020 28584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 22:30:37.209338 28599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:37.210726 28584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 22:30:37.210889 28584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal
uuid: "f12059c2e8e84a0a9c934b87eb31254b"
format_stamp: "Formatted at 2026-06-19 22:30:37 on dist-test-slave-bt27"
server_key: "b7f011bc4b05bc8bbb63263e5528bb3a"
server_key_iv: "26463e4a29b9333137a542c90ac043b3"
server_key_version: "encryptionkey@0"
I20260619 22:30:37.211004 28584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:37.224025 28584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:37.227466 28584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:37.227676 28584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:37.234606 28584 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.253:45903
I20260619 22:30:37.234910 28651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.253:45903 every 8 connection(s)
I20260619 22:30:37.235579 28584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/data/info.pb
I20260619 22:30:37.238359 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:37.242273 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28584
I20260619 22:30:37.242427 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-1/wal/instance
I20260619 22:30:37.242295 28652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.242745 5827 external_mini_cluster.cc:1428] Setting key 9dda3b96612f96a191490c147f029110
WARNING: no policy specified for kudu/127.5.176.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.176.252@KRBTEST.COM" created.
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:37.259899 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.244475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.5.176.254:33937)
0619 22:30:37.244651 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.244657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.245557 (+ 900us) server_negotiation.cc:407] Connection header received
0619 22:30:37.246586 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.246589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.246652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.246760 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.247900 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.248433 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.249580 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.249823 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.249940 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.254968 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.254975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.254981 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.255021 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.257033 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.257821 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.257826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.257844 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.257949 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.258440 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.258447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.258450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.258713 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.258829 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.259566 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.259703 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 22:30:37.260742 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.244837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33937 (local address 127.0.0.1:34892)
0619 22:30:37.245386 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.245428 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.246381 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.246926 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.246933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.247289 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.247753 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.247766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.248566 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.248569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.249449 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.249459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.249984 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.249988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.250225 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.250832 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.250848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.254749 (+ 3901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.257172 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.257179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.257236 (+ 57us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.257651 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.258066 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.258072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.258077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.258319 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.258892 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.258900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.259274 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.259486 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.259852 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":128,"threads_started":1}
I20260619 22:30:37.262413 28652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.264670 28577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } attempt: 3
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab.
Entry for principal kudu/127.5.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab.
I20260619 22:30:37.273191 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.263100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.5.176.253:45903)
0619 22:30:37.263431 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.263441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.263462 (+ 21us) server_negotiation.cc:407] Connection header received
0619 22:30:37.263506 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.263510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.263664 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.263812 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.264159 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.265246 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.266519 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.267009 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.267132 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.269381 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.269392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.269408 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.269436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.271287 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.271694 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.271700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.271702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.271784 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.272055 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.272058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.272060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.272350 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.272496 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.272764 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.272905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":153,"threads_started":1}
I20260619 22:30:37.273399 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.262950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40516)
0619 22:30:37.263068 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.263083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.263160 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.263782 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.263785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.263960 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.264052 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.264057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.265390 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.265396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.266276 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.266287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.267126 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.267130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.267237 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.267771 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.267785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.269242 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.271388 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.271394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.271397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.271586 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.271879 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.271882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.271883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.271965 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.272456 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.272459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.272535 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.272656 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.272826 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 22:30:37.273921 5827 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4QvkS_/build/debug/bin/kudu
/tmp/dist-test-task4QvkS_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:33165
--webserver_interface=127.5.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab
--principal=kudu/127.5.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.176.212:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.176.254:33937,127.5.176.253:45903,127.5.176.252:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 22:30:37.275296 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.265097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40530)
0619 22:30:37.265200 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.265216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.265300 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.265727 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.265730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.265899 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.265992 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.265998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.266766 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.266769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.267951 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.267961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.268499 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.268502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.268621 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.269010 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.269021 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.270716 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.273572 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.273576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.273578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.273864 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.274190 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.274193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.274195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.274281 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.274665 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.274668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.274774 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.274883 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.275119 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 22:30:37.275321 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.265134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.5.176.253:45903)
0619 22:30:37.265485 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.265491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.265500 (+ 9us) server_negotiation.cc:407] Connection header received
0619 22:30:37.265555 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.265558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.265604 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.265706 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.266105 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.266652 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.268070 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.268385 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.268492 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.270871 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.270874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.270877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.270906 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.273448 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.273978 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.273981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.273982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.274059 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.274389 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.274392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.274393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.274581 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.274676 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.274942 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.275154 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":129,"threads_started":1}
I20260619 22:30:37.275955 28652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.276374 28577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.276688 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.276415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39898)
0619 22:30:37.276525 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.276597 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 22:30:37.276857 28601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
I20260619 22:30:37.277122 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.276844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39908)
0619 22:30:37.276984 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.277045 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 22:30:37.277661 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.277443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39910)
0619 22:30:37.277550 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.277597 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 22:30:37.277806 28652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:33165: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
I20260619 22:30:37.278053 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.277697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39924)
0619 22:30:37.277920 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.277975 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260619 22:30:37.278245 28577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:33165: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
I20260619 22:30:37.309422 28577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } attempt: 1
I20260619 22:30:37.310341 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.309983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39936)
0619 22:30:37.310172 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.310243 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 22:30:37.311038 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.310824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39946)
0619 22:30:37.310925 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.310979 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 22:30:37.311183 28577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:33165: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
I20260619 22:30:37.327072 28652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } attempt: 1
I20260619 22:30:37.328011 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.327651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39948)
0619 22:30:37.327838 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.327913 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 22:30:37.328832 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.328589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39964)
0619 22:30:37.328706 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.328758 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 22:30:37.328999 28652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:33165: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
I20260619 22:30:37.386358 28577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } attempt: 2
I20260619 22:30:37.387253 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.386921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39968)
0619 22:30:37.387083 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.387157 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 22:30:37.387988 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.387775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39976)
0619 22:30:37.387878 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.387928 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 22:30:37.388147 28577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:33165: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
W20260619 22:30:37.388813 28659 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 22:30:37.389108 28659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 22:30:37.389158 28659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 22:30:37.389263 28659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 22:30:37.393455 28659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 22:30:37.393514 28659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 22:30:37.393534 28659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 22:30:37.393601 28659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 22:30:37.393635 28659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 22:30:37.397190 28652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } attempt: 2
I20260619 22:30:37.397985 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.397672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39988)
0619 22:30:37.397832 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.397898 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 22:30:37.398687 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.398486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:39996)
0619 22:30:37.398586 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.398634 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 22:30:37.398828 28652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.176.252:33165: Network error: Client connection negotiation failed: client connection to 127.5.176.252:33165: connect: Connection refused (error 111)
I20260619 22:30:37.399611 28659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.176.212:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.176.254:33937,127.5.176.253:45903,127.5.176.252:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/krb5kdc/kudu_127.5.176.252.keytab
--principal=kudu/127.5.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.176.252:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 21:32:11 UTC on ccycloud.quasar-utnohq.root.comops.site
build id 4456676
I20260619 22:30:37.400692 28659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 22:30:37.401691 28659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 22:30:37.410166 28668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:37.410108 28665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 22:30:37.410099 28666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 22:30:37.411183 28659 server_base.cc:1034] running on GCE node
I20260619 22:30:37.411926 28659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 22:30:37.413442 28659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 22:30:37.414611 28659 hybrid_clock.cc:648] HybridClock initialized: now 1781908237414546 us; error 79 us; skew 500 ppm
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 22:30:37.418802 28659 init.cc:377] Logged in from keytab as kudu/127.5.176.252@KRBTEST.COM (short username kudu)
I20260619 22:30:37.420035 28659 webserver.cc:460] Webserver started at http://127.5.176.252:40651/ using document root <none> and password file <none>
I20260619 22:30:37.420830 28659 fs_manager.cc:359] Metadata directory not provided
I20260619 22:30:37.420956 28659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 22:30:37.421234 28659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 22:30:37.424422 28659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data/instance:
uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241"
format_stamp: "Formatted at 2026-06-19 22:30:37 on dist-test-slave-bt27"
server_key: "6544132b7f9b80f777e3ea57e2afbbf3"
server_key_iv: "487767504ff3de4b7e10cfa2e0db9e06"
server_key_version: "encryptionkey@0"
I20260619 22:30:37.425165 28659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal/instance:
uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241"
format_stamp: "Formatted at 2026-06-19 22:30:37 on dist-test-slave-bt27"
server_key: "6544132b7f9b80f777e3ea57e2afbbf3"
server_key_iv: "487767504ff3de4b7e10cfa2e0db9e06"
server_key_version: "encryptionkey@0"
I20260619 22:30:37.430018 28659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 22:30:37.433459 28675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 22:30:37.435263 28659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 22:30:37.435410 28659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data,/tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal
uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241"
format_stamp: "Formatted at 2026-06-19 22:30:37 on dist-test-slave-bt27"
server_key: "6544132b7f9b80f777e3ea57e2afbbf3"
server_key_iv: "487767504ff3de4b7e10cfa2e0db9e06"
server_key_version: "encryptionkey@0"
I20260619 22:30:37.435547 28659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 22:30:37.453905 28659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 22:30:37.458545 28659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 22:30:37.458787 28659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 22:30:37.466662 28659 rpc_server.cc:307] RPC server started. Bound to: 127.5.176.252:33165
I20260619 22:30:37.466953 28727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.176.252:33165 every 8 connection(s)
I20260619 22:30:37.467716 28659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/data/info.pb
I20260619 22:30:37.470614 28728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 22:30:37.472059 5827 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4QvkS_/build/debug/bin/kudu as pid 28659
I20260619 22:30:37.472199 5827 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4QvkS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781907951519384-5827-0/minicluster-data/master-2/wal/instance
I20260619 22:30:37.472500 5827 external_mini_cluster.cc:1428] Setting key 4f6e390155b1aadd5dc9c07dc88591d9
I20260619 22:30:37.474979 28728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
I20260619 22:30:37.487306 28577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } attempt: 3
I20260619 22:30:37.487708 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.476848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.5.176.254:33937)
0619 22:30:37.477006 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.477010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.477336 (+ 326us) server_negotiation.cc:407] Connection header received
0619 22:30:37.477494 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.477496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.477543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.477611 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.478450 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.479221 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.480468 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.480732 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.480791 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.483743 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.483749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.483752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.483783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.485292 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.486163 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.486168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.486171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.486262 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.486580 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.486584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.486586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.486781 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.486901 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.487429 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.487555 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 22:30:37.488481 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.476790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.254:33937 (local address 127.0.0.1:34898)
0619 22:30:37.477196 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.477244 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.477374 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.477682 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.477687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.478110 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.478335 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.478341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.479365 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.479368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.480343 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.480351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.480939 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.480941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.481017 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.481492 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.481508 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.483555 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.485726 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.485733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.485737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.486030 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.486393 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.486396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.486397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.486498 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.486892 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.486899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.487051 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.487366 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.487556 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":35,"thread_start_us":148,"threads_started":1}
I20260619 22:30:37.488482 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.473918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.5.176.252:33165)
0619 22:30:37.474438 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.474452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.474485 (+ 33us) server_negotiation.cc:407] Connection header received
0619 22:30:37.475149 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.475164 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.475576 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.475903 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.476712 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.478317 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.479503 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.480212 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.480410 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.483215 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.483224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.483237 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.483267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.485283 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.485712 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.485716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.485721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.485836 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.486154 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.486160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.486164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.486612 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.486846 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.487066 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.487396 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":408,"spinlock_wait_cycles":3456,"thread_start_us":155,"threads_started":1}
I20260619 22:30:37.489727 28728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.489987 28652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } attempt: 3
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.254@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.254@KRBTEST.COM
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.253@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
I20260619 22:30:37.497668 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.487699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:40020)
0619 22:30:37.487950 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.487965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.488035 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.488431 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.488435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.488613 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.488722 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.488729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.489813 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.489817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.490588 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.490602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.491194 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.491197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.491283 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.492196 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.492214 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.493734 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.495899 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.495902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.495904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.496157 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.496495 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.496498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.496500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.496585 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.497078 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.497084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.497201 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.497300 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.497458 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260619 22:30:37.498415 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.487924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.5.176.252:33165)
0619 22:30:37.488195 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.488204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.488217 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:30:37.488248 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.488251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.488291 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.488833 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.488941 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.489691 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.490725 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.491065 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.493854 (+ 2789us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.493970 (+ 116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.493973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.493975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.493991 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.495759 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.496281 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.496284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.496286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.496380 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.496704 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.496707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.496708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.496958 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.497983 (+ 1025us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.498067 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.498219 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260619 22:30:37.500728 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.490343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.5.176.253:45903)
0619 22:30:37.490467 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.490471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.490611 (+ 140us) server_negotiation.cc:407] Connection header received
0619 22:30:37.490781 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.490784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.490827 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.490871 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.491427 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.492183 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.493311 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.493711 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.493801 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.496869 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.496873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.496876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.496900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.498605 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.499240 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.499244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.499245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.499329 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.499697 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.499704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.499706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.499976 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.500082 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.500416 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.500547 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260619 22:30:37.501502 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.490334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.253:45903 (local address 127.0.0.1:40544)
0619 22:30:37.490547 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.490560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.490676 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.490993 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.490996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.491177 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.491278 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.491284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.492318 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.492320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.493168 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.493184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.494660 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.494667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.494795 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.495229 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.495244 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.496727 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.498887 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.498890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.498892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.499123 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.499471 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.499473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.499475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.499570 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.500095 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.500101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.500205 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.501115 (+ 910us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.501316 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260619 22:30:37.502045 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.491674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.5.176.254:33937)
0619 22:30:37.491776 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.491780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.491792 (+ 12us) server_negotiation.cc:407] Connection header received
0619 22:30:37.491822 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.491825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.491859 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.491945 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.492751 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.493434 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.494235 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.494583 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.494674 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.498424 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.498427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.498430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.498453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.500065 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.500551 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.500554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.500556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.500633 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.500974 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.500979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.500983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.501177 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.501383 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.501774 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.501908 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:30:37.502902 28577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: Bootstrap starting.
I20260619 22:30:37.502997 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.490371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:40026)
0619 22:30:37.490524 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.490537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.490612 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.491372 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.491375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.491558 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.491634 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.491639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.493081 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.493084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.494468 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.494479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.495626 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.495630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.495733 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.496200 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.496219 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.498779 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.501465 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.501468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.501470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.501705 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.502009 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.502012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.502014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.502097 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.502559 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.502562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.502650 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.502728 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.502848 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":539}
I20260619 22:30:37.503654 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.490826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.5.176.252:33165)
0619 22:30:37.491136 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.491142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.491155 (+ 13us) server_negotiation.cc:407] Connection header received
0619 22:30:37.491190 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.491192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.491241 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.491718 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.491787 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.492964 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.494575 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.495482 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.496777 (+ 1295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.498901 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.498908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.498914 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.499296 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.501342 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.501832 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.501835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.501836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.501911 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.502250 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.502252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.502254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.502449 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.503330 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.503401 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.503504 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":247,"spinlock_wait_cycles":768}
I20260619 22:30:37.505873 28577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:37.506094 28728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
I20260619 22:30:37.506850 28652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: Bootstrap starting.
I20260619 22:30:37.507678 28577 log.cc:824] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: Log is configured to *not* fsync() on all Append() calls
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.176.253@KRBTEST.COM
I20260619 22:30:37.509430 28652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:37.509941 28577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: No bootstrap required, opened a new log
I20260619 22:30:37.510241 28652 log.cc:824] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:37.512218 28652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: No bootstrap required, opened a new log
I20260619 22:30:37.513264 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.503600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.5.176.253:45903)
0619 22:30:37.503701 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.503705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.503761 (+ 56us) server_negotiation.cc:407] Connection header received
0619 22:30:37.503891 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.503894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.503934 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.504019 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.504784 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.505276 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.506485 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.506866 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.506943 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.509573 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.509577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.509579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.509604 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.511407 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.511875 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.511879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.511881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.511962 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.512228 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.512231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.512246 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.512462 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.512629 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.512974 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.513084 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 22:30:37.514170 28577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.514386 28577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90dd1260ba514226a61e454611d6db46, State: Initialized, Role: FOLLOWER
I20260619 22:30:37.514729 28577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
Jun 19 22:30:37 dist-test-slave-bt27 krb5kdc[28496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781908237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.176.252@KRBTEST.COM for kudu/127.5.176.252@KRBTEST.COM
I20260619 22:30:37.516306 28577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:37.516469 28652 raft_consensus.cc:348] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.516749 28652 raft_consensus.cc:729] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12059c2e8e84a0a9c934b87eb31254b, State: Initialized, Role: FOLLOWER
I20260619 22:30:37.517081 5827 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 22:30:37.517135 28530 tablet.cc:2369] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 22:30:37.517107 28652 consensus_queue.cc:260] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.518596 28652 sys_catalog.cc:564] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:37.518435 28551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "90dd1260ba514226a61e454611d6db46"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908
I20260619 22:30:37.519117 28551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 22:30:37.519245 28551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:37.520692 28551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.520766 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.506583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.176.252:33165 (local address 127.0.0.1:40034)
0619 22:30:37.508159 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0619 22:30:37.508174 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 22:30:37.508249 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 22:30:37.511112 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 22:30:37.511116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 22:30:37.511294 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 22:30:37.511388 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.511395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.512363 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.512366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.513134 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 22:30:37.513144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.513783 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.513787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 22:30:37.513864 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.514394 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 22:30:37.514411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 22:30:37.516041 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 22:30:37.518750 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.518755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.518759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.518985 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.519747 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 22:30:37.519750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 22:30:37.519752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 22:30:37.519824 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 22:30:37.520222 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 22:30:37.520225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 22:30:37.520314 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 22:30:37.520412 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 22:30:37.520556 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501}
I20260619 22:30:37.521119 28736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.521234 28736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:37.521678 28551 leader_election.cc:288] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [CANDIDATE]: Term 1 election: Requested vote from peers f12059c2e8e84a0a9c934b87eb31254b (127.5.176.253:45903), 6efcfafad87e4ecbabcd0c7a9ae2e241 (127.5.176.252:33165)
I20260619 22:30:37.522346 28626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90dd1260ba514226a61e454611d6db46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f12059c2e8e84a0a9c934b87eb31254b"
I20260619 22:30:37.522559 28626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:37.523897 28626 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90dd1260ba514226a61e454611d6db46 in term 1.
I20260619 22:30:37.524886 28527 leader_election.cc:302] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90dd1260ba514226a61e454611d6db46, f12059c2e8e84a0a9c934b87eb31254b; no voters:
W20260619 22:30:37.525079 28760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:37.525148 28760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:37.525346 28742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.525483 28742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:30:37.526479 28748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:37.526590 28748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:37.526640 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.506799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.5.176.252:33165)
0619 22:30:37.510397 (+ 3598us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.510797 (+ 400us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.510814 (+ 17us) server_negotiation.cc:407] Connection header received
0619 22:30:37.510859 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.510865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.510919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.511005 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.511510 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.512254 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.513244 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.513554 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.513654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.516275 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.516279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.516281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.516305 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.518623 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.519559 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.519565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.519569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.519645 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.519908 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.519911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.519912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.520110 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.520193 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.520951 (+ 758us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.526490 (+ 5539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3503}
I20260619 22:30:37.526849 28736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 22:30:37.527064 28736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [term 1 LEADER]: Becoming Leader. State: Replica: 90dd1260ba514226a61e454611d6db46, State: Running, Role: LEADER
I20260619 22:30:37.527431 28736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.526113 28584 master_runner.cc:186] Error getting master registration for 127.5.176.252:33165: OK, instance_id { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" instance_seqno: 1781908237464541 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:30:37.529460 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90dd1260ba514226a61e454611d6db46. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.529580 28584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:30:37.529596 28728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: Bootstrap starting.
I20260619 22:30:37.529650 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:37.531898 28728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: Neither blocks nor log segments found. Creating new log.
I20260619 22:30:37.532750 28728 log.cc:824] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: Log is configured to *not* fsync() on all Append() calls
I20260619 22:30:37.535027 28728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: No bootstrap required, opened a new log
I20260619 22:30:37.533632 28509 master_runner.cc:186] Error getting master registration for 127.5.176.252:33165: OK, instance_id { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" instance_seqno: 1781908237464541 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 22:30:37.537082 28509 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 22:30:37.538462 28768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 22:30:37.539173 28728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.539444 28728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efcfafad87e4ecbabcd0c7a9ae2e241, State: Initialized, Role: FOLLOWER
I20260619 22:30:37.539835 28728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } }
I20260619 22:30:37.540200 28768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 22:30:37.541481 28728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 22:30:37.541567 28702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90dd1260ba514226a61e454611d6db46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241"
I20260619 22:30:37.541791 28702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [term 0 FOLLOWER]: Advancing to term 1
I20260619 22:30:37.543798 28702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90dd1260ba514226a61e454611d6db46 in term 1.
I20260619 22:30:37.552991 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.538866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.5.176.254:33937)
0619 22:30:37.538982 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.538986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.539536 (+ 550us) server_negotiation.cc:407] Connection header received
0619 22:30:37.539864 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.539868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.539924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.539982 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.544431 (+ 4449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.545156 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.546327 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.546624 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.546701 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.549409 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.549413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.549415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.549443 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.551261 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.551693 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.551697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.551699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.551787 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.552067 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.552071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.552073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.552272 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.552424 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.552690 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.552825 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 22:30:37.553900 28702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [term 1 FOLLOWER]: Refusing update from remote peer 90dd1260ba514226a61e454611d6db46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:37.553938 28626 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [term 1 FOLLOWER]: Refusing update from remote peer 90dd1260ba514226a61e454611d6db46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 22:30:37.554576 28736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:37.555016 28736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 22:30:37.555505 28773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.555629 28773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:37.558951 28742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90dd1260ba514226a61e454611d6db46. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.559091 28742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 22:30:37.561549 28785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 22:30:37.561643 28785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 22:30:37.561789 28773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90dd1260ba514226a61e454611d6db46. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.561899 28773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:37.566732 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.539479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.5.176.252:33165)
0619 22:30:37.539602 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.539606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.544722 (+ 5116us) server_negotiation.cc:407] Connection header received
0619 22:30:37.544907 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.544910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.544967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.545019 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.553584 (+ 8565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.554313 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.556535 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.556908 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.557013 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.561765 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.561769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.561772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.561802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.563709 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.565343 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.565347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.565349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.565435 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.565716 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.565719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.565721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.565947 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.566063 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.566439 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.566568 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 22:30:37.566851 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:37.538762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.5.176.253:45903)
0619 22:30:37.538928 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.538933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.546298 (+ 7365us) server_negotiation.cc:407] Connection header received
0619 22:30:37.546481 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.546485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.546536 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.546590 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.547531 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.548172 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.557845 (+ 9673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:37.558212 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:37.558304 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:37.562159 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 22:30:37.562164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 22:30:37.562167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 22:30:37.562199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 22:30:37.564501 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.565315 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.565322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.565323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.565428 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 22:30:37.565783 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 22:30:37.565786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 22:30:37.565788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 22:30:37.566028 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 22:30:37.566131 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:37.566555 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 22:30:37.566683 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 22:30:37.573945 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.574116 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:37.574281 28774 mvcc.cc:205] Tried to move back new op lower bound from 7298696141013020672 to 7298696140916043776. Current Snapshot: MvccSnapshot[applied={T|T < 7298696141013020672}]
I20260619 22:30:37.576258 28773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.576399 28773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:37.576689 28736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.576831 28736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46 [sys.catalog]: This master's current role is: LEADER
I20260619 22:30:37.578325 28742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90dd1260ba514226a61e454611d6db46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90dd1260ba514226a61e454611d6db46" member_type: VOTER last_known_addr { host: "127.5.176.254" port: 33937 } } peers { permanent_uuid: "f12059c2e8e84a0a9c934b87eb31254b" member_type: VOTER last_known_addr { host: "127.5.176.253" port: 45903 } } peers { permanent_uuid: "6efcfafad87e4ecbabcd0c7a9ae2e241" member_type: VOTER last_known_addr { host: "127.5.176.252" port: 33165 } } }
I20260619 22:30:37.578454 28742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 22:30:37.580092 28787 mvcc.cc:205] Tried to move back new op lower bound from 7298696141013020672 to 7298696140916043776. Current Snapshot: MvccSnapshot[applied={T|T < 7298696141013020672}]
I20260619 22:30:37.584517 28768 catalog_manager.cc:1347] Generated new cluster ID: 06ff6c0cdc024878b5fd7c9ce9c5dc12
I20260619 22:30:37.584599 28768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 22:30:37.608706 28768 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 22:30:37.610052 28768 catalog_manager.cc:1504] Loading token signing keys...
I20260619 22:30:37.626387 28768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90dd1260ba514226a61e454611d6db46: Generated new TSK 0
I20260619 22:30:37.627518 28768 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 22:30:37.676795 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.675137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.5.176.253:45903)
0619 22:30:37.675265 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.675269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.675612 (+ 343us) server_negotiation.cc:407] Connection header received
0619 22:30:37.675852 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.675856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.675910 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.675960 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.676637 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40574: BlockingRecv error: recv got EOF from 127.0.0.1:40574 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 22:30:37.678002 28581 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.674611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.5.176.254:33937)
0619 22:30:37.674748 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.674753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.677206 (+ 2453us) server_negotiation.cc:407] Connection header received
0619 22:30:37.677313 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.677316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.677373 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.677471 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.677882 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34934: BlockingRecv error: recv got EOF from 127.0.0.1:34934 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 22:30:37.679273 28731 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.676015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.5.176.252:33165)
0619 22:30:37.676148 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.676154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.678402 (+ 2248us) server_negotiation.cc:407] Connection header received
0619 22:30:37.678574 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.678578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.678627 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.678676 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.679156 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40042: BlockingRecv error: recv got EOF from 127.0.0.1:40042 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 22:30:37.686533 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.684766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.5.176.253:45903)
0619 22:30:37.684895 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.684899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.685456 (+ 557us) server_negotiation.cc:407] Connection header received
0619 22:30:37.685675 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.685679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.685758 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.685823 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.686422 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40576: BlockingRecv error: recv got EOF from 127.0.0.1:40576 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 22:30:37.692528 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.690957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.5.176.253:45903)
0619 22:30:37.691064 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.691068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.691597 (+ 529us) server_negotiation.cc:407] Connection header received
0619 22:30:37.691797 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.691801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.691854 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.691906 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.692418 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40580: BlockingRecv error: recv got EOF from 127.0.0.1:40580 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 22:30:37.707749 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.698343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.5.176.253:45903)
0619 22:30:37.698470 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.698474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.706678 (+ 8204us) server_negotiation.cc:407] Connection header received
0619 22:30:37.706935 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.706939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.707003 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.707094 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.707627 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40584: BlockingRecv error: recv got EOF from 127.0.0.1:40584 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 22:30:37.724560 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.722673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.5.176.253:45903)
0619 22:30:37.722835 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.722840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.723478 (+ 638us) server_negotiation.cc:407] Connection header received
0619 22:30:37.723692 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.723697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.723751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.723810 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.724421 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40594: BlockingRecv error: recv got EOF from 127.0.0.1:40594 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 22:30:37.737299 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.735474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.5.176.253:45903)
0619 22:30:37.735613 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.735620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.736241 (+ 621us) server_negotiation.cc:407] Connection header received
0619 22:30:37.736473 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.736480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.736535 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.736588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.737154 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40610: BlockingRecv error: recv got EOF from 127.0.0.1:40610 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 22:30:37.757602 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.755944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.5.176.253:45903)
0619 22:30:37.756064 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.756068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.756633 (+ 565us) server_negotiation.cc:407] Connection header received
0619 22:30:37.756831 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.756835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.756887 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.756938 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.757480 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40624: BlockingRecv error: recv got EOF from 127.0.0.1:40624 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 22:30:37.792986 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.791936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.5.176.253:45903)
0619 22:30:37.792109 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.792112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.792163 (+ 51us) server_negotiation.cc:407] Connection header received
0619 22:30:37.792285 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.792288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.792337 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.792415 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.792839 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40632: BlockingRecv error: recv got EOF from 127.0.0.1:40632 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 22:30:37.860191 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.859105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.5.176.253:45903)
0619 22:30:37.859285 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.859288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.859307 (+ 19us) server_negotiation.cc:407] Connection header received
0619 22:30:37.859475 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.859477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.859529 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.859602 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.860059 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40636: BlockingRecv error: recv got EOF from 127.0.0.1:40636 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 22:30:37.991849 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:37.990534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.5.176.253:45903)
0619 22:30:37.990698 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:37.990704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:37.990918 (+ 214us) server_negotiation.cc:407] Connection header received
0619 22:30:37.991109 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:37.991112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:37.991167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:37.991262 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:37.991740 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40650: BlockingRecv error: recv got EOF from 127.0.0.1:40650 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 22:30:38.251235 28658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 22:30:38.250053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.5.176.253:45903)
0619 22:30:38.250221 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:38.250225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:38.250331 (+ 106us) server_negotiation.cc:407] Connection header received
0619 22:30:38.250502 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:38.250506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:38.250561 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:38.250650 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 22:30:38.251123 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40664: BlockingRecv error: recv got EOF from 127.0.0.1:40664 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 22:30:38.526568 28760 catalog_manager.cc:1259] Loaded cluster ID: 06ff6c0cdc024878b5fd7c9ce9c5dc12
I20260619 22:30:38.526661 28760 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: loading cluster ID for follower catalog manager: success
I20260619 22:30:38.528358 28760 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: acquiring CA information for follower catalog manager: success
I20260619 22:30:38.529457 28760 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f12059c2e8e84a0a9c934b87eb31254b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:30:38.563026 28785 catalog_manager.cc:1259] Loaded cluster ID: 06ff6c0cdc024878b5fd7c9ce9c5dc12
I20260619 22:30:38.563124 28785 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: loading cluster ID for follower catalog manager: success
I20260619 22:30:38.564826 28785 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: acquiring CA information for follower catalog manager: success
I20260619 22:30:38.566022 28785 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6efcfafad87e4ecbabcd0c7a9ae2e241: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 22:30:38.769377 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:38.765347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.5.176.253:45903)
0619 22:30:38.765602 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:38.765609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:38.765702 (+ 93us) server_negotiation.cc:407] Connection header received
0619 22:30:38.765881 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:38.765884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:38.766019 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:38.766136 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:38.766429 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.766971 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.767850 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.768088 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.768199 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:38.768523 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:38.768979 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:38.769116 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:38.769153 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 22:30:38.769210 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260619 22:30:38.776338 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:38.772447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.5.176.252:33165)
0619 22:30:38.772669 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:38.772676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:38.772711 (+ 35us) server_negotiation.cc:407] Connection header received
0619 22:30:38.772914 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:38.772917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:38.773085 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:38.773217 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:38.773444 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.773983 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.774847 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.775080 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.775193 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:38.775500 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:38.775948 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:38.776079 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:38.776119 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 22:30:38.776176 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260619 22:30:38.783649 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:38.779240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.5.176.253:45903)
0619 22:30:38.779388 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:38.779392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:38.779814 (+ 422us) server_negotiation.cc:407] Connection header received
0619 22:30:38.779979 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:38.779982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:38.780051 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:38.780157 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:38.780703 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.781459 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.782431 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.782776 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.782880 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:38.783136 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:38.783214 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:38.783292 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:38.783417 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 22:30:38.783471 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 22:30:38.783706 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:38.779754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.5.176.252:33165)
0619 22:30:38.779873 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:38.779878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:38.779973 (+ 95us) server_negotiation.cc:407] Connection header received
0619 22:30:38.780087 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:38.780093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:38.780144 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:38.780205 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:38.780586 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.781123 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.782308 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.782639 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.782733 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:38.783034 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:38.783118 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:38.783199 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:38.783531 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 22:30:38.783586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 22:30:38.784826 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 22:30:38.779191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.5.176.254:33937)
0619 22:30:38.779461 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 22:30:38.779466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 22:30:38.779584 (+ 118us) server_negotiation.cc:407] Connection header received
0619 22:30:38.779821 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 22:30:38.779825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 22:30:38.780003 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 22:30:38.780148 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 22:30:38.780528 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.781385 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.782562 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 22:30:38.782907 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 22:30:38.783009 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 22:30:38.783591 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 22:30:38.784190 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 22:30:38.784277 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 22:30:38.784587 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 22:30:38.784659 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260619 22:30:38.787843 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28509
I20260619 22:30:38.794476 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28584
I20260619 22:30:38.800702 5827 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4QvkS_/build/debug/bin/kudu with pid 28659
2026-06-19T22:30:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3149 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6308 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287271 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST